Premium
METHODOLOGICAL AND PSYCHOMETRIC ISSUES IN SETTING CUTOFF SCORES USING THE ANGOFF METHOD
Author(s) -
MAURER TODD J.,
ALEXANDER RALPH A.,
CALLAHAN CATHY M.,
BAILEY JEFF J.,
DAMBROT FAYE H.
Publication year - 1991
Publication title -
personnel psychology
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 6.076
H-Index - 142
eISSN - 1744-6570
pISSN - 0031-5826
DOI - 10.1111/j.1744-6570.1991.tb00958.x
Subject(s) - psychology , normative , reliability (semiconductor) , consistency (knowledge bases) , set (abstract data type) , identification (biology) , inter rater reliability , quality (philosophy) , internal consistency , process (computing) , test (biology) , applied psychology , social psychology , statistics , computer science , psychometrics , clinical psychology , artificial intelligence , mathematics , rating scale , developmental psychology , philosophy , power (physics) , physics , botany , operating system , paleontology , epistemology , quantum mechanics , biology , programming language
The present study examined several psychometric issues relevant to the use of a favored technique (the Angoff method) used to set standards in criterion referenced testing. The research was conducted within a setting which allowed (a) confident identification of expert and non‐expert judges, and (b) estimation of “true” scores for items judged so that accuracy of judgments in addition to reliability could be examined. Results suggested that expertise of judges does make a difference in producing more accurate and reliable data, underscoring the importance of using true subject matter experts (SMEs) in the judgment process. A rater analysis technique (rater‐total correlations) was illustrated, which might prove useful in improving the quality of data obtained using the Angoff method, particularly when there is some question regarding the internal consistency of ratings and expertise of some of the raters. Finally, a rater accuracy adjustment/calibration technique was examined and proved to be a potentially useful method to maximize accuracy of a standard derived using the Angoff method in settings where archival normative test data can be obtained. Other methods that could potentially be used to improve Angoff data were discussed.