Open AccessAN EXAMINATION OF THE LINK BETWEEN RATER CALIBRATION PERFORMANCE AND SUBSEQUENT SCORING ACCURACY IN GRADUATE RECORD EXAMINATIONS ® (GRE ® ) WRITING
Author(s) -
RickerPedley Kathryn L.
Publication year - 2011
Publication title -
ets research report series
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.235
H-Index - 5
ISSN - 2330-8516
DOI - 10.1002/j.2333-8504.2011.tb02239.x
Subject(s) - calibration , correlation , set (abstract data type) , statistics , test (biology) , computer science , inter rater reliability , medical physics , mathematics , reliability engineering , psychology , medicine , rating scale , paleontology , geometry , engineering , biology , programming language
A pseudo‐experimental study was conducted to examine the link between rater accuracy calibration performances and subsequent accuracy during operational scoring. The study asked 45 raters to score a 75‐response calibration set and then a 100‐response (operational) set of responses from a retired Graduate Record Examinations ® (GRE ® ) writing prompt that had been vetted to determine correct scores prior to the study. The study found a positive relationship between calibration accuracy and scoring accuracy. Results suggest that a longer calibration test is more reliable and has a stronger correlation with operational scoring exact agreement; however, even tests as short as 10 responses had a reasonable correlation with operational accuracy, and the shorter test classified raters accurately 87% of the time when compared to the 75‐response criterion. The additional requirement of no discrepant scores during calibration had better correlation with exact agreement, but much poorer classification accuracy in the shorter tests. This research suggests that calibration is useful for screening raters in its current form, but that further investigation of passing standards and of less controlled operational scoring conditions is required.