English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

Human annotated data is the cornerstone of today's artificial intelligenceefforts, yet data labeling processes can be complicated and expensive,especially when human labelers disagree with each other. The current workpractice is to use majority-voted labels to overrule the disagreement. However,in the subjective data labeling tasks such as hate speech annotation,disagreement among individual labelers can be difficult to resolve. In thispaper, we explored why such disagreements occur using a mixed-method approach -including interviews with experts, concept mapping exercises, andself-reporting items - to develop a multidimensional scale for distilling theprocess of how annotators label a hate speech corpus. We tested this scale with170 annotators in a hate speech annotation task. Results showed that our scalecan reveal facets of individual differences among annotators (e.g., age,personality, etc.), and these facets' relationships to an annotator's finallabel decision of an instance. We suggest that this work contributes to theunderstanding of how humans annotate data. The proposed scale can potentiallyimprove the value of the currently discarded minority-vote labels.

The Origin and Value of Disagreement Among Data Labelers: A Case Study of the Individual Difference in Hate Speech Annotation