English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

When training data are collected from human annotators, the design of theannotation instrument, the instructions given to annotators, thecharacteristics of the annotators, and their interactions can impact trainingdata. This study demonstrates that design choices made when creating anannotation instrument also impact the models trained on the resultingannotations. We introduce the term annotation sensitivity to refer to theimpact of annotation data collection methods on the annotations themselves andon downstream model performance and predictions. We collect annotations of hatespeech and offensive language in five experimental conditions of an annotationinstrument, randomly assigning annotators to conditions. We then fine-tune BERTmodels on each of the five resulting datasets and evaluate model performance ona holdout portion of each condition. We find considerable differences betweenthe conditions for 1) the share of hate speech/offensive language annotations,2) model performance, 3) model predictions, and 4) model learning curves. Ourresults emphasize the crucial role played by the annotation instrument whichhas received little attention in the machine learning literature. We call foradditional research into how and why the instrument impacts the annotations toinform the development of best practices in instrument design.

Annotation Sensitivity: Training Data Collection Methods Affect Model Performance