Premium
Real‐Time Feedback on Rater Drift in Constructed‐Response Items: An Example From the Golden State Examination
Author(s) -
Hoskens Machteld,
Wilson Mark
Publication year - 2001
Publication title -
journal of educational measurement
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.917
H-Index - 47
eISSN - 1745-3984
pISSN - 0022-0655
DOI - 10.1111/j.1745-3984.2001.tb01119.x
Subject(s) - psychology , statistics , test (biology) , social psychology , control (management) , econometrics , mathematics , computer science , artificial intelligence , paleontology , biology
In this study, patterns of variation in severities of a group of raters over time or so‐called “rater drift” was examined when raters scored an essay written under examination conditions. At the same time feedback was given to rater leaders (called “table leaders”) who then interpreted the feedback and reported to the raters. Rater severities in five successive periods were estimated using a modified linear logistic test model (LLTM, Fischer, 1973) approach. It was found that the raters did indeed drift towards the mean, but a planned comparision of the feedback with a control condition was not successful; it was believed that this was due to contamination at the table leader level. A series of models was also estimated designed to detect other types of rater effects beyond severity: a tendency to use extreme scores, and tendency to prefer certain categories. The models for these effects were found to be showing significant improvement in fit, implying that these effects were indeed present, although they were difficult to detect in relatively short time periods.