Premium
An Experimental Study of the Internal Consistency of Judgments Made in Bookmark Standard Setting
Author(s) -
Clauser Brian E.,
Baldwin Peter,
Margolis Melissa J.,
Mee Janet,
Winward Marcia
Publication year - 2017
Publication title -
journal of educational measurement
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.917
H-Index - 47
eISSN - 1745-3984
pISSN - 0022-0655
DOI - 10.1111/jedm.12157
Subject(s) - consistency (knowledge bases) , internal consistency , context (archaeology) , psychology , internal validity , social psychology , computer science , psychometrics , statistics , artificial intelligence , clinical psychology , mathematics , paleontology , biology
Validating performance standards is challenging and complex. Because of the difficulties associated with collecting evidence related to external criteria, validity arguments rely heavily on evidence related to internal criteria—especially evidence that expert judgments are internally consistent. Given its importance, it is somewhat surprising that evidence of this kind has rarely been published in the context of the widely used bookmark standard‐setting procedure. In this article we examined the effect of ordered item booklet difficulty on content experts’ bookmark judgments. If panelists make internally consistent judgments, their resultant cut scores should be unaffected by the difficulty of their respective booklets. This internal consistency was not observed: the results suggest that substantial systematic differences in the resultant cut scores can arise when the difficulty of the ordered item booklets varies. These findings raise questions about the ability of content experts to make the judgments required by the bookmark procedure.