Open AccessCOMPUTER ANALYSIS OF ESSAY CONTENT FOR AUTOMATED SCORE PREDICTION: A PROTOTYPE AUTOMATED SCORING SYSTEM FOR GMAT ANALYTICAL WRITING ASSESSMENT ESSAYS
Author(s) -
Burstein Jill,
BradenHarder Lisa,
Chodorow Martin,
Hua Shuyi,
Kaplan Bruce,
Kukich Karen,
Lu Chi,
Nolan James,
Rock Don,
Wolff Susanne
Publication year - 1998
Publication title -
ets research report series
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.235
H-Index - 5
ISSN - 2330-8516
DOI - 10.1002/j.2333-8504.1998.tb01764.x
Subject(s) - rubric , argument (complex analysis) , set (abstract data type) , computer science , feature (linguistics) , writing assessment , variable (mathematics) , artificial intelligence , natural language processing , variables , linear regression , machine learning , mathematics education , psychology , linguistics , mathematics , programming language , mathematical analysis , biochemistry , chemistry , philosophy
This report discusses the development and evaluation of a research prototype system designed to automatically score essay responses to the GMAT Analytical Writing Assessments: (a). Analysis of an Argument (Argument essays) and (b). Analysis of an Issue (Issue essays) item types. The system, Electronic Essay Rater ( e‐rater ), was designed to automatically analyze several features of an essay and score the essay based on the features of writing as specified in holistic rubrics. E‐rater uses a hybrid feature methodology. It incorporates several variables that are derived statistically, extracted through NLP techniques, or achieved by simple “counting” procedures. The version of the e‐rater described in this report uses five sets of critical feature variables to build the final linear regression model used for predicting scores. The same set of critical variables was used to fit models for the issue and argument training essays and the following results were achieved. For the set of 275 cross‐validation data, exact or adjacent agreement with human rater scores reached 95%. For the 282 cross‐validation issue essays exact or adjacent agreement with human rater scores achieved 93%. The rich feature variables used as score predictors in e‐rater could potentially be used to generate explanation of score predictions, and diagnostic and instructional information.