Premium
An Examination of Rater Drift Within a Generalizability Theory Framework
Author(s) -
Harik Polina,
Clauser Brian E.,
Grabovsky Irina,
Nungester Ronald J.,
Swanson Dave,
Nandakumar Ratna
Publication year - 2009
Publication title -
journal of educational measurement
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.917
H-Index - 47
eISSN - 1745-3984
pISSN - 0022-0655
DOI - 10.1111/j.1745-3984.2009.01068.x
Subject(s) - generalizability theory , psychology , statistics , econometrics , stability (learning theory) , function (biology) , mathematics , developmental psychology , computer science , machine learning , evolutionary biology , biology
The present study examined the long‐term usefulness of estimated parameters used to adjust the scores from a performance assessment to account for differences in rater stringency. Ratings from four components of the USMLE® Step 2 Clinical Skills Examination data were analyzed. A generalizability‐theory framework was used to examine the extent to which rater‐related sources of error could be eliminated through statistical adjustment. Particular attention was given to the stability of these estimated parameters over time. The results suggest that rater stringency estimates obtained at a point in time and then used to adjust ratings over a period of months may substantially decrease in usefulness. In some cases, over several months, the use of these adjustments may become counterproductive. Additionally, it is hypothesized that the rate of deterioration in the usefulness of estimated parameters may be a function of the characteristics of the scale.