We found a match
Your institution may have access to this item. Find your institution then sign in to continue.
- Title
Crowdsourcing with unsure option.
- Authors
Ding, Yao-Xiang; Zhou, Zhi-Hua
- Abstract
One of the fundamental issues in crowdsourcing is the trade-off between the number of workers needed for high-accuracy aggregation and the budget to pay. To save cost, it is important to ensure high quality of the crowd-sourced labels, hence the total cost on label collection will be reduced. Since the confidence of the workers often has a close relationship with their abilities, a possible way for quality control is to request the workers to return the labels only when they feel confident, by means of providing them with an ‘unsure’ option. On the other hand, allowing workers to choose the unsure option can potentially waste part of the budget. In this work, we conduct an analysis towards understanding when providing the unsure option indeed leads to significant cost reduction, as well as how the confidence threshold might be set. We also propose an online mechanism, which is an alternative for threshold selection when the estimation of the crowd ability distribution is difficult.
- Subjects
CROWDSOURCING; DATA mining; DATA science; MACHINE learning; AUTOMATIC extracting (Information science); CLASSIFICATION algorithms
- Publication
Machine Learning, 2018, Vol 107, Issue 4, p749
- ISSN
0885-6125
- Publication type
Article
- DOI
10.1007/s10994-017-5677-x