Premium
Effect of Content Knowledge on Angoff‐Style Standard Setting Judgments
Author(s) -
Margolis Melissa J.,
Mee Janet,
Clauser Brian E.,
Winward Marcia,
Clauser Jerome C.
Publication year - 2016
Publication title -
educational measurement: issues and practice
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.158
H-Index - 52
eISSN - 1745-3992
pISSN - 0731-1745
DOI - 10.1111/emip.12104
Subject(s) - credibility , psychology , test (biology) , context (archaeology) , set (abstract data type) , selection (genetic algorithm) , style (visual arts) , argument (complex analysis) , perception , social psychology , content validity , applied psychology , computer science , psychometrics , medicine , artificial intelligence , clinical psychology , paleontology , archaeology , history , neuroscience , political science , law , biology , programming language
Evidence to support the credibility of standard setting procedures is a critical part of the validity argument for decisions made based on tests that are used for classification. One area in which there has been limited empirical study is the impact of standard setting judge selection on the resulting cut score. One important issue related to judge selection is whether the extent of judges’ content knowledge impacts their perceptions of the probability that a minimally proficient examinee will answer the item correctly. The present article reports on two studies conducted in the context of Angoff‐style standard setting for medical licensing examinations. In the first study, content experts answered and subsequently provided Angoff judgments for a set of test items. After accounting for perceived item difficulty and judge stringency, answering the item correctly accounted for a significant (and potentially important) impact on expert judgment. The second study examined whether providing the correct answer to the judges would result in a similar effect to that associated with knowing the correct answer. The results suggested that providing the correct answer did not impact judgments. These results have important implications for the validity of standard setting outcomes in general and on judge recruitment specifically.