Premium
Using a Difficulty‐Anchored Rating Scale in Performing Angoff Ratings
Author(s) -
Hoffman Calvin C.,
Tashima C. Chy,
Luck Gypsi
Publication year - 2010
Publication title -
international journal of selection and assessment
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.812
H-Index - 61
eISSN - 1468-2389
pISSN - 0965-075X
DOI - 10.1111/j.1468-2389.2010.00522.x
Subject(s) - normative , rating scale , psychology , test (biology) , scale (ratio) , social psychology , applied psychology , developmental psychology , political science , paleontology , physics , quantum mechanics , biology , law
While the Angoff (1971) is a commonly used cut score method, critics (Berk, 1996; Impara & Plake, 1997) argue the Angoff places too‐high cognitive demands on raters. In response to criticisms of the Angoff, a number of modifications to the method have been proposed. Some suggested Angoff modifications include using an iterative rating process, presenting judges with normative data about item performance, revising the rating judgment into a Yes/No decision, assigning relative weights to dimensions within a test, and using item response theory in setting cut scores. In this study, subject matter expert raters were provided with a ‘difficulty anchored’ rating scale to use while making Angoff ratings; this scale can be viewed as a variation of the Angoff normative data modification. The rating scale presented test items having known p ‐values as anchors, and served as a simple means of providing normative information to guide the Angoff rating process. Results are discussed regarding reliability of the mean Angoff rating (.73) and the correlation of mean Angoff ratings with item difficulty (observed r ranges from .65 to .73).