Premium
Least Squares Models to Correct for Rater Effects in Performance Assessment
Author(s) -
Raymond Mark R.,
Viswesvaran Chockalingam
Publication year - 1993
Publication title -
journal of educational measurement
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.917
H-Index - 47
eISSN - 1745-3984
pISSN - 0022-0655
DOI - 10.1111/j.1745-3984.1993.tb00426.x
Subject(s) - ordinary least squares , statistics , generalized least squares , partial least squares regression , inter rater reliability , least squares function approximation , logistic regression , non linear least squares , mathematics , econometrics , psychology , reliability (semiconductor) , explained sum of squares , rating scale , power (physics) , physics , quantum mechanics , estimator
This study describes three least squares models to control for rater effects in performance evaluation: ordinary least squares (OLS); weighted least squares (WLS); and ordinary least squares, subsequent to applying a logistic transformation to observed ratings (LOG‐OLS). The models were applied to ratings obtained from four administrations of an oral examination required for certification in a medical specialty. For any single administration, there were 40 raters and approximately 115 candidates, and each candidate was rated by four raters. The results indicated that raters exhibited significant amounts of leniency error and that application of the least squares models would change the pass‐fail status of approximately 7% to 9% of the candidates. Ratings adjusted by the models demonstrated higher reliability and correlated slightly higher than observed ratings with the scores on a written examination.