English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

With the rapid growth of crowdsourcing platforms it has become easy andrelatively inexpensive to collect a dataset labeled by multiple annotators in ashort time. However due to the lack of control over the quality of theannotators, some abnormal annotators may be affected by position bias which canpotentially degrade the quality of the final consensus labels. In this paper weintroduce a statistical framework to model and detect annotator's position biasin order to control the false discovery rate (FDR) without a prior knowledge onthe amount of biased annotators - the expected fraction of false discoveriesamong all discoveries being not too high, in order to assure that most of thediscoveries are indeed true and replicable. The key technical developmentrelies on some new knockoff filters adapted to our problem and new algorithmsbased on the Inverse Scale Space dynamics whose discretization is potentiallysuitable for large scale crowdsourcing data analysis. Our studies are supportedby experiments with both simulated examples and real-world data. The proposedframework provides us a useful tool for quantitatively studying annotator'sabnormal behavior in crowdsourcing data arising from machine learning,sociology, computer vision, multimedia, etc.

False Discovery Rate Control and Statistical Quality Assessment of Annotators in Crowdsourced Ranking