Premium
Rater Certification Tests: A Psychometric Approach
Author(s) -
Attali Yigal
Publication year - 2019
Publication title -
educational measurement: issues and practice
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.158
H-Index - 52
eISSN - 1745-3992
pISSN - 0731-1745
DOI - 10.1111/emip.12248
Subject(s) - certification , reliability (semiconductor) , test (biology) , applied psychology , quality (philosophy) , psychology , psychometric testing , psychometrics , inter rater reliability , scale (ratio) , computer science , item response theory , reliability engineering , rating scale , clinical psychology , cronbach's alpha , engineering , developmental psychology , political science , paleontology , power (physics) , philosophy , physics , epistemology , quantum mechanics , law , biology
Rater training is an important part of developing and conducting large‐scale constructed‐response assessments. As part of this process, candidate raters have to pass a certification test to confirm that they are able to score consistently and accurately before they begin scoring operationally. Moreover, many assessment programs require raters to pass a calibration test before every scoring shift. To support the high‐stakes decisions made on the basis of rater certification tests, a psychometric approach for their development, analysis, and use is proposed. The circumstances and uses of these tests suggest that they are expected to have relatively low reliability. This expectation is supported by empirical data. Implications for the development and use of these tests to ensure their quality are discussed.