Open AccessADJUSTMENT FOR READER RATING BEHAVIOR IN THE TEST OF WRITTEN ENGLISH
Author(s) -
Longford Nicholas T.
Publication year - 1995
Publication title -
ets research report series
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.235
H-Index - 5
ISSN - 2330-8516
DOI - 10.1002/j.2333-8504.1995.tb01673.x
Subject(s) - test (biology) , quality (philosophy) , statistics , psychology , econometrics , scheme (mathematics) , computer science , process (computing) , mathematics , paleontology , mathematical analysis , philosophy , epistemology , biology , operating system
This report evaluates the impact of a potential scheme for score adjustment using data from the administrations of the Test of Written English in 1994. It is shown that, assuming non‐informative assignment of readers to essays, the adjustment due to reader differences reduces the mean squared error for all essays except those graded by readers with small workloads. The quality of the rating process, as described by the variances due to true scores, severity, and inconsistency, as well as the distribution of workloads, are similar across the administrations. This would make a reliable prediction of the optimal score adjustment in future administrations possible. Two approximations to the optimal adjustment are proposed.