Open AccessA DIFFERENTIAL WORD USE MEASURE FOR CONTENT ANALYSIS IN AUTOMATED ESSAY SCORING
Author(s) -
Attali Yigal
Publication year - 2011
Publication title -
ets research report series
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.235
H-Index - 5
ISSN - 2330-8516
DOI - 10.1002/j.2333-8504.2011.tb02272.x
Subject(s) - task (project management) , vocabulary , measure (data warehouse) , context (archaeology) , content (measure theory) , natural language processing , word (group theory) , computer science , content analysis , differential (mechanical device) , psychology , artificial intelligence , cognitive psychology , statistics , linguistics , mathematics , data mining , paleontology , mathematical analysis , philosophy , social science , management , sociology , biology , engineering , economics , aerospace engineering
This paper proposes an alternative content measure for essay scoring, based on the difference in the relative frequency of a word in high‐scored versus low‐scored essays. The differential word use (DWU) measure is the average of these differences across all words in the essay. A positive value indicates the essay is using vocabulary more typical of high‐scoring essays in this task, and vice versa. In addition to the traditional prompt level, this measure is also computed at the generic task level, where the content of an essay is evaluated in the context of the general vocabulary of the writing task (e.g., GRE ® issue), across different prompts. Evaluation results across four GRE and TOEFL ® tasks are presented. Factor analyses show that both the prompt and task DWU measures load on the same factor as the prompt‐specific content analysis measures of e‐rater ® . Regression results on the human essay scores show that the generic task DWU measure is a strong predictor of human scores, second only to essay length among noncontent e‐rater features. This measure provides a way to conduct a task‐level content analysis that is related to the prompt‐specific content analysis of e‐rater but does not require prompt‐specific training. Regression results for the prompt‐level DWU show that it can be used as a replacement to prompt‐specific content‐vector analysis (CVA) features.