Open AccessSTUDIES OF A LATENT CLASS SIGNAL DETECTION MODEL FOR CONSTRUCTED RESPONSE SCORING II: INCOMPLETE AND HIERARCHICAL DESIGNS
Author(s) -
DeCarlo Lawrence T.
Publication year - 2010
Publication title -
ets research report series
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.235
H-Index - 5
ISSN - 2330-8516
DOI - 10.1002/j.2333-8504.2010.tb02215.x
Subject(s) - class (philosophy) , latent class model , item response theory , nested set model , psychology , multilevel model , hierarchical database model , scale (ratio) , computer science , signal (programming language) , artificial intelligence , statistics , econometrics , natural language processing , machine learning , data mining , mathematics , developmental psychology , psychometrics , physics , quantum mechanics , relational database , programming language
A basic consideration in large‐scale assessments that use constructed response (CR) items, such as essays, is how to allocate the essays to the raters that score them. Designs that are used in practice are incomplete, in that each essay is scored by only a subset of the raters, and also unbalanced, in that the number of essays scored by each rater differs across the raters. In addition, all of the possible rater pairs may not be used. The present study examines the effects of these factors on parameter recovery and classification accuracy using simulations of a latent class model based on signal detection theory (SDT). Many tests also include more than one CR item, which introduces a nested or hierarchical structure into the design, in that raters are nested within essays (i.e., there are multiple raters per essay) and essays are nested within examinees (i.e., each examinee provides two or more essays). A hierarchical rater model (HRM) has previously been developed to recognize the nested structure. A version of the HRM that incorporates a latent class signal detection model in the first level, referred to as the HRM‐SDT model, is presented. Parameter recovery in the HRM‐SDT model is examined in simulations. The model is applied to data from several ETS tests.