English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Zendy Plus

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Zendy Tools

Zendy Open

Machine learning (ML) and artificial intelligence (AI) systems rely heavilyon human-annotated data for training and evaluation. A major challenge in thiscontext is the occurrence of annotation errors, as their effects can degrademodel performance. This paper presents a predictive error model trained todetect potential errors in search relevance annotation tasks for threeindustry-scale ML applications (music streaming, video streaming, and mobileapps). Drawing on real-world data from an extensive search relevance annotationprogram, we demonstrate that errors can be predicted with moderate modelperformance (AUC=0.65-0.75) and that model performance generalizes well acrossapplications (i.e., a global, task-agnostic model performs on par withtask-specific models). In contrast to past research, which has often focused onpredicting annotation labels from task-specific features, our model is trainedto predict errors directly from a combination of task features and behavioralfeatures derived from the annotation process, in order to achieve a high degreeof generalizability. We demonstrate the usefulness of the model in the contextof auditing, where prioritizing tasks with high predicted error probabilitiesconsiderably increases the amount of corrected annotation errors (e.g., 40%efficiency gains for the music streaming application). These results highlightthat behavioral error detection models can yield considerable improvements inthe efficiency and quality of data annotation processes. Our findings revealcritical insights into effective error management in the data annotationprocess, thereby contributing to the broader field of human-in-the-loop ML.

Generalizable Error Modeling for Human Data Annotation: Evidence From an Industry-Scale Search Data Annotation Program