Premium
The Choice of Response Probability in Bookmark Standard Setting: An Experimental Study
Author(s) -
Baldwin Peter,
Margolis Melissa J.,
Clauser Brian E.,
Mee Janet,
Winward Marcia
Publication year - 2019
Publication title -
educational measurement: issues and practice
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.158
H-Index - 52
eISSN - 1745-3992
pISSN - 0731-1745
DOI - 10.1111/emip.12230
Subject(s) - consistency (knowledge bases) , internal consistency , argument (complex analysis) , computer science , multiple choice , psychology , statistics , artificial intelligence , mathematics , psychometrics , medicine , significant difference
Evidence of the internal consistency of standard‐setting judgments is a critical part of the validity argument for tests used to make classification decisions. The bookmark standard‐setting procedure is a popular approach to establishing performance standards, but there is relatively little research that reflects on the internal consistency of the resulting judgments. This article presents the results of an experiment in which content experts were randomly assigned to one of two response probability conditions: .67 and .80. If the standard‐setting judgments collected with the bookmark procedure are internally consistent, both conditions should produce highly similar cut scores. The results showed substantially different cut scores for the two conditions; this calls into question whether content experts can produce the type of internally consistent judgments that are required using the bookmark procedure.