Open AccessCALIBRATION OF ESSAY READERS FINAL REPORT
Author(s) -
Braun Henry I.
Publication year - 1986
Publication title -
ets research report series
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.235
H-Index - 5
ISSN - 2330-8516
DOI - 10.1002/j.2330-8516.1986.tb00164.x
Subject(s) - reliability (semiconductor) , calibration , computer science , variation (astronomy) , raw score , block (permutation group theory) , statistics , estimation , reliability engineering , econometrics , raw data , mathematics , geometry , quantum mechanics , power (physics) , physics , astrophysics , engineering , management , economics
Scoring reliability of essays and other free response questions is of considerable concern. This report describes a statistically designed experiment that was carried out in an operational setting to determine the contributions of different sources of variation to the unreliability of scoring. The experiment made novel use of partially balanced incomplete block designs that facilitated the unbiased estimation of certain main effects without requiring readers to assess the same paper several times. In addition, estimates were obtained of the improvement in reliability that result from removing variability from systematic sources of variation by an appropriate adjustment of the raw scores. This statistical calibration appears to be a cost‐effective approach to enhancing scoring reliability when compared to simply increasing the number of readings per paper. The results of the experiment also provide a framework for examining other, simpler calibration strategies. One such strategy is briefly examined.