Premium
Reliability and Attribute‐Based Scoring in Cognitive Diagnostic Assessment
Author(s) -
Gierl Mark J.,
Cui Ying,
Zhou Jiawen
Publication year - 2009
Publication title -
journal of educational measurement
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.917
H-Index - 47
eISSN - 1745-3984
pISSN - 0022-0655
DOI - 10.1111/j.1745-3984.2009.00082.x
Subject(s) - reliability (semiconductor) , variance (accounting) , cognition , set (abstract data type) , strengths and weaknesses , computer science , test (biology) , sample (material) , psychology , statistics , natural language processing , artificial intelligence , mathematics , social psychology , paleontology , power (physics) , chemistry , physics , accounting , chromatography , quantum mechanics , neuroscience , business , biology , programming language
The attribute hierarchy method (AHM) is a psychometric procedure for classifying examinees’ test item responses into a set of structured attribute patterns associated with different components from a cognitive model of task performance. Results from an AHM analysis yield information on examinees’ cognitive strengths and weaknesses. Hence, the AHM can be used for cognitive diagnostic assessment. The purpose of this study is to introduce and evaluate a new concept for assessing attribute reliability using the ratio of true score variance to observed score variance on items that probe specific cognitive attributes. This reliability procedure is evaluated and illustrated using both simulated data and student response data from a sample of algebra items taken from the March 2005 administration of the SAT. The reliability of diagnostic scores and the implications for practice are also discussed.