English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

In recent years, progress in NLU has been driven by benchmarks. Thesebenchmarks are typically collected by crowdsourcing, where annotators writeexamples based on annotation instructions crafted by dataset creators. In thiswork, we hypothesize that annotators pick up on patterns in the crowdsourcinginstructions, which bias them to write many similar examples that are thenover-represented in the collected data. We study this form of bias, termedinstruction bias, in 14 recent NLU benchmarks, showing that instructionexamples often exhibit concrete patterns, which are propagated by crowdworkersto the collected data. This extends previous work (Geva et al., 2019) andraises a new concern of whether we are modeling the dataset creator'sinstructions, rather than the task. Through a series of experiments, we showthat, indeed, instruction bias can lead to overestimation of model performance,and that models struggle to generalize beyond biases originating in thecrowdsourcing instructions. We further analyze the influence of instructionbias in terms of pattern frequency and model size, and derive concreterecommendations for creating future NLU benchmarks.

Don't Blame the Annotator: Bias Already Starts in the Annotation Instructions