English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Online crowdsourcing provides a scalable and inexpensive means to collectknowledge (e.g. labels) about various types of data items (e.g. text, audio,video). However, it is also known to result in large variance in the quality ofrecorded responses which often cannot be directly used for training machinelearning systems. To resolve this issue, a lot of work has been conducted tocontrol the response quality such that low-quality responses cannot adverselyaffect the performance of the machine learning systems. Such work is referredto as the quality control for crowdsourcing. Past quality control research canbe divided into two major branches: quality control mechanism design andstatistical models. The first branch focuses on designing measures, thresholds,interfaces and workflows for payment, gamification, question assignment andother mechanisms that influence workers' behaviour. The second branch focuseson developing statistical models to perform effective aggregation of responsesto infer correct responses. The two branches are connected as statisticalmodels (i) provide parameter estimates to support the measure and thresholdcalculation, and (ii) encode modelling assumptions used to derive (theoretical)performance guarantees for the mechanisms. There are surveys regarding eachbranch but they lack technical details about the other branch. Our survey isthe first to bridge the two branches by providing technical details on how theywork together under frameworks that systematically unify crowdsourcing aspectsmodelled by both of them to determine the response quality. We are also thefirst to provide taxonomies of quality control papers based on the proposedframeworks. Finally, we specify the current limitations and the correspondingfuture directions for the quality control research.

A Technical Survey on Statistical Modelling and Design Methods for Crowdsourcing Quality Control