English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Human variation in labeling is often considered noise. Annotation projectsfor machine learning (ML) aim at minimizing human label variation, with theassumption to maximize data quality and in turn optimize and maximize machinelearning metrics. However, this conventional practice assumes that there existsa ground truth, and neglects that there exists genuine human variation inlabeling due to disagreement, subjectivity in annotation or multiple plausibleanswers. In this position paper, we argue that this big open problem of humanlabel variation persists and critically needs more attention to move our fieldforward. This is because human label variation impacts all stages of the MLpipeline: data, modeling and evaluation. However, few works consider all ofthese dimensions jointly; and existing research is fragmented. We reconciledifferent previously proposed notions of human label variation, provide arepository of publicly-available datasets with un-aggregated labels, depictapproaches proposed so far, identify gaps and suggest ways forward. As datasetsare becoming increasingly available, we hope that this synthesized view on the'problem' will lead to an open discussion on possible strategies to devisefundamentally new directions.

The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation