Premium
An Examination of the Replicability of Angoff Standard Setting Results Within a Generalizability Theory Framework
Author(s) -
Clauser Jerome C.,
Margolis Melissa J.,
Clauser Brian E.
Publication year - 2014
Publication title -
journal of educational measurement
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.917
H-Index - 47
eISSN - 1745-3984
pISSN - 0022-0655
DOI - 10.1111/jedm.12038
Subject(s) - generalizability theory , argument (complex analysis) , assertion , variance (accounting) , panel data , facet (psychology) , econometrics , standard error , stability (learning theory) , psychology , statistics , computer science , social psychology , mathematics , medicine , economics , machine learning , accounting , personality , big five personality traits , programming language
Evidence of stable standard setting results over panels or occasions is an important part of the validity argument for an established cut score. Unfortunately, due to the high cost of convening multiple panels of content experts, standards often are based on the recommendation from a single panel of judges. This approach implicitly assumes that the variability across panels will be modest, but little evidence is available to support this assertion. This article examines the stability of Angoff standard setting results across panels. Data were collected for six independent standard setting exercises, with three panels participating in each exercise. The results show that although in some cases the panel effect is negligible, for four of the six data sets the panel facet represented a large portion of the overall error variance. Ignoring the often hidden panel/occasion facet can result in artificially optimistic estimates of the cut score stability. Results based on a single panel should not be viewed as a reasonable estimate of the results that would be found over multiple panels. Instead, the variability seen in a single panel can best be viewed as a lower bound of the expected variability when the exercise is replicated.