Open AccessAnnotation Difficulties in Natural Language Inference
Author(s) -
Aikaterini-Lida Kalouli,
Livy Real,
Annebeth Buis,
Martha Palmer,
Valéria de Paiva
Publication year - 2021
Language(s) - English
Resource type - Conference proceedings
DOI - 10.5753/stil.2021.17804
Subject(s) - computer science , inference , task (project management) , natural language processing , annotation , mainstream , artificial intelligence , quality (philosophy) , natural (archaeology) , natural language , epistemology , philosophy , theology , management , archaeology , economics , history
State-of-the-art models have obtained high accuracy on mainstream Natural Language Inference (NLI) datasets. However, recent research has suggested that the task is far from solved. Current models struggle to generalize and fail to consider the inherent human disagreements in tasks such as NLI. In this work, we conduct an experiment based on a small subset of the NLI corpora such as SNLI and SICK. It reveals that some inference cases are inherently harder to annotate than others, although good-quality guidelines can reduce this difficulty to some extent. We propose adding a Difficulty Score to NLI datasets, to capture the human difficulty level of agreement.