English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Crowdsourcing has been part of the IR toolbox as a cheap and fast mechanismto obtain labels for system development and evaluation. Successful deploymentof crowdsourcing at scale involves adjusting many variables, a very importantone being the number of workers needed per human intelligence task (HIT). Weconsider the crowdsourcing task of learning the answer to simplemultiple-choice HITs, which are representative of many relevance experiments.In order to provide statistically significant results, one often needs to askmultiple workers to answer the same HIT. A stopping rule is an algorithm that,given a HIT, decides for any given set of worker answers if the system shouldstop and output an answer or iterate and ask one more worker. Knowing thehistoric performance of a worker in the form of a quality score can bebeneficial in such a scenario. In this paper we investigate how to devisebetter stopping rules given such quality scores. We also suggest adaptiveexploration as a promising approach for scalable and automatic creation ofground truth. We conduct a data analysis on an industrial crowdsourcingplatform, and use the observations from this analysis to design new stoppingrules that use the workers' quality scores in a non-trivial manner. We thenperform a simulation based on a real-world workload, showing that our algorithmperforms better than the more naive approaches.

How Many Workers to Ask? Adaptive Exploration for Collecting High Quality Labels