A TREE‐BASED APPROACH TO PROFICIENCY SCALING

The process of translating examinees' observed test results into probabilistic statements about the patterns of skill mastery associated with performance at successive points on a test's reported score scale has been termed proficiency scaling . This paper introduces a new approach to proficiency scaling which was specifically developed to provide accurate, instructionally‐relevant score interpretations, even when the test under consideration is a non‐diagnostic, broad‐based ability test such as the SAT. The approach employs techniques derived from the realm of tree‐based regression to classify test items into nonoverlapping clusters such that all of the items in each cluster have similar values of item difficulty and require similar combinations of skills. A nonparametric smoothing procedure is then used to summarize cluster performance conditional on underlying test score. The resulting cluster characteristic curves provide the probability of responding correctly to items with specified skill requirements, expressed as a function of the underlying test score. This new approach is described in the context of a specific application: developing proficiency interpretations for the reading comprehension subsection of the SAT I Verbal Reasoning Test. It is shown that the generated proficiency interpretations account for 91% of the variation which would be accounted for by an optimal clustering solution. These results suggest that the approach is capable of extracting useful information about the specific combinations of skills needed to perform at increasingly higher levels on a test's reported score scale.