Open AccessImplementing a Contributory Scoring Approach for the GRE ® Analytical Writing Section: A Comprehensive Empirical Investigation
Author(s) -
Breyer F. Jay,
Rupp André A.,
Bridgeman Brent
Publication year - 2017
Publication title -
ets research report series
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.235
H-Index - 5
ISSN - 2330-8516
DOI - 10.1002/ets2.12142
Subject(s) - computer science , task (project management) , test (biology) , argument (complex analysis) , quality (philosophy) , section (typography) , test score , empirical research , psychology , data science , machine learning , artificial intelligence , applied psychology , management science , mathematics education , statistics , standardized test , mathematics , epistemology , engineering , paleontology , biochemistry , chemistry , philosophy , biology , operating system , systems engineering
In this research report, we present an empirical argument for the use of a contributory scoring approach for the 2‐essay writing assessment of the analytical writing section of the GRE ® test in which human and machine scores are combined for score creation at the task and section levels. The approach was designed to replace a currently operational all‐human check scoring approach in which machine scores are used solely as quality‐control checks to determine when additional human ratings are needed due to unacceptably large score discrepancies. We use data from 6 samples of essays collected from test takers during operational administrations and special validity studies to empirically evaluate 6 different score computation methods. During the presentation of our work, we critically discuss key methodological design decisions and underlying rationales for these decisions. We close the report by discussing how the research methodology is generalizable to other testing programs and use contexts.