English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Cognitive psychologists have documented that humans use cognitive heuristics,or mental shortcuts, to make quick decisions while expending less effort. Whileperforming annotation work on crowdsourcing platforms, we hypothesize that suchheuristic use among annotators cascades on to data quality and modelrobustness. In this work, we study cognitive heuristic use in the context ofannotating multiple-choice reading comprehension datasets. We propose trackingannotator heuristic traces, where we tangibly measure low-effort annotationstrategies that could indicate usage of various cognitive heuristics. We findevidence that annotators might be using multiple such heuristics, based oncorrelations with a battery of psychological tests. Importantly, heuristic useamong annotators determines data quality along several dimensions: (1) knownbiased models, such as partial input models, more easily solve examplesauthored by annotators that rate highly on heuristic use, (2) models trained onannotators scoring highly on heuristic use don't generalize as well, and (3)heuristic-seeking annotators tend to create qualitatively less challengingexamples. Our findings suggest that tracking heuristic usage among annotatorscan potentially help with collecting challenging datasets and diagnosing modelbiases.

Cascading Biases: Investigating the Effect of Heuristic Annotation Strategies on Data and Models