English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Crowdsourcing has emerged as an effective platform for labeling large amountsof data in a cost- and time-efficient manner. Most previous work has focused ondesigning an efficient algorithm to recover only the ground-truth labels of thedata. In this paper, we consider multi-choice crowdsourcing tasks with the goalof recovering not only the ground truth, but also the most confusing answer andthe confusion probability. The most confusing answer provides usefulinformation about the task by revealing the most plausible answer other thanthe ground truth and how plausible it is. To theoretically analyze suchscenarios, we propose a model in which there are the top two plausible answersfor each task, distinguished from the rest of the choices. Task difficulty isquantified by the probability of confusion between the top two, and workerreliability is quantified by the probability of giving an answer among the toptwo. Under this model, we propose a two-stage inference algorithm to infer boththe top two answers and the confusion probability. We show that our algorithmachieves the minimax optimal convergence rate. We conduct both synthetic andreal data experiments and demonstrate that our algorithm outperforms otherrecent algorithms. We also show the applicability of our algorithms ininferring the difficulty of tasks and in training neural networks with top-twosoft labels.

Recovering Top-Two Answers and Confusion Probability in Multi-Choice Crowdsourcing