Getting More From Your Data: Application Of Item Response Theory To The Statistics Concept Inventory

This paper applies the techniques of Item Response Theory (IRT) to data from the Statistics Concept Inventory (SCI). Based on results from the Fall 2005 post-test ( n = 422 students), the analyses of IRT are compared with those of Classical Test Theory (CTT). The concepts are extended to discussions of other applications, such as computerized adaptive testing and Likert-scale items which may be of interest to the engineering education community. While techniques based on CTT generally yield valuable information, methods of IRT can reveal unanticipated subtleties in a dataset. For example, items of extreme difficulty (hard or easy) typically attain low discrimination indices (CTT), thus labeling them as “poor”. Application of IRT can identify these items as strongly discriminating among students of extreme ability (high or low). The three simplest IRT models (one-, two-, and three-parameter) are compared to illustrate cases where they differ. The theoretical foundations of IRT are provided, extending to validating the assumptions for the SCI dataset and discussing other potential uses of IRT that are applicable to survey design in engineering education.