We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

stat.ML

Change to browse by:

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Machine Learning

Title: False Discovery Rate Control and Statistical Quality Assessment of Annotators in Crowdsourced Ranking

Abstract: With the rapid growth of crowdsourcing platforms it has become easy and relatively inexpensive to collect a dataset labeled by multiple annotators in a short time. However due to the lack of control over the quality of the annotators, some abnormal annotators may be affected by position bias which can potentially degrade the quality of the final consensus labels. In this paper we introduce a statistical framework to model and detect annotator's position bias in order to control the false discovery rate (FDR) without a prior knowledge on the amount of biased annotators - the expected fraction of false discoveries among all discoveries being not too high, in order to assure that most of the discoveries are indeed true and replicable. The key technical development relies on some new knockoff filters adapted to our problem and new algorithms based on the Inverse Scale Space dynamics whose discretization is potentially suitable for large scale crowdsourcing data analysis. Our studies are supported by experiments with both simulated examples and real-world data. The proposed framework provides us a useful tool for quantitatively studying annotator's abnormal behavior in crowdsourcing data arising from machine learning, sociology, computer vision, multimedia, etc.
Comments: ICML 2016 accepted
Subjects: Machine Learning (stat.ML)
Cite as: arXiv:1605.05860 [stat.ML]
  (or arXiv:1605.05860v3 [stat.ML] for this version)

Submission history

From: Qianqian Xu [view email]
[v1] Thu, 19 May 2016 09:14:39 GMT (1705kb,D)
[v2] Mon, 6 Jun 2016 10:23:44 GMT (1705kb,D)
[v3] Thu, 16 Jun 2016 15:30:43 GMT (1705kb,D)

Link back to: arXiv, form interface, contact.