Open AccessHolistic Discourse Coherence Annotation for Noisy Essay Writing
Author(s) -
Jill Burstein,
Joel Tetreault,
Martin Chodorow
Publication year - 2013
Publication title -
dialogue and discourse
Language(s) - English
Resource type - Journals
ISSN - 2152-9620
DOI - 10.5087/dad.2013.202
Subject(s) - annotation , coherence (philosophical gambling strategy) , computer science , natural language processing , noisy data , artificial intelligence , taxonomy (biology) , scheme (mathematics) , mathematics , statistics , botany , biology , mathematical analysis
This paper reviews annotation schemes used for labeling discourse coherence in well-formed and noisy (essay) data, and it describes a system that we have developed for automated holistic scoring of essay coherence. We review previous, related work on unsupervised computational approaches to evaluating discourse coherence and focus on a taxonomy of discourse coherence schemes classified by their different goals and types of data. We illustrate how a holistic approach can be successfully used to build systems for noisy essay data, across domains and populations. We discuss the model features related to human scoring guide criteria for essay scoring, and the importance of using model features relevant to these criteria for the purpose of generating meaningful scores and feedback for students and test-takers. To demonstrate the effectiveness of a holistic annotation scheme, we present results of system evaluations.