Premium
Using text classification to improve annotation quality by improving annotator consistency
Author(s) -
Ishita Emi,
Fukuda Satoshi,
Tomiura Yoichi,
Oard Douglas W.
Publication year - 2020
Publication title -
proceedings of the association for information science and technology
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.193
H-Index - 14
ISSN - 2373-9231
DOI - 10.1002/pra2.301
Subject(s) - annotation , computer science , classifier (uml) , operationalization , consistency (knowledge bases) , limiting , natural language processing , artificial intelligence , information retrieval , quality (philosophy) , task (project management) , mechanical engineering , philosophy , management , epistemology , engineering , economics
This paper presents results of experiments in which annotators were asked to selectively reexamine their decisions when those decisions seemed inconsistent. The annotation task was binary topic classification. To operationalize the concept of annotation consistency, a text classifier was trained on all manual annotations made during a complete first pass and then used to automatically recode every document. Annotators were then asked to perform a second manual pass, limiting their attention to cases in which their first annotation disagreed with the text classifier. On average across three annotators, each working independently, 11% of first pass annotations were reconsidered, 46% of reconsidered annotations were changed in the second pass, and 71% of changed annotations agreed with decisions made independently by an experienced fourth annotator. The net result was that for an 11% average increase in annotation cost it was possible to increase overall chance corrected agreement with the annotation decisions of an experienced annotator (as measured by kappa) from 0.70 to 0.75.