Prediction of true test scores from observed item scores and ancillary data

In many educational tests which involve constructed responses, a traditional test score is obtained by adding together item scores obtained through holistic scoring by trained human raters. For example, this practice was used until 2008 in the case of GRE ® G eneral A nalytical W riting and until 2009 in the case of TOEFL ® i BT W riting. With use of natural language processing, it is possible to obtain additional information concerning item responses from computer programs such as e‐rater ® . In addition, available information relevant to examinee performance may include scores on related tests. We suggest application of standard results from classical test theory to the available data to obtain best linear predictors of true traditional test scores. In performing such analysis, we require estimation of variances and covariances of measurement errors, a task which can be quite difficult in the case of tests with limited numbers of items and with multiple measurements per item. As a consequence, a new estimation method is suggested based on samples of examinees who have taken an assessment more than once. Such samples are typically not random samples of the general population of examinees, so that we apply statistical adjustment methods to obtain the needed estimated variances and covariances of measurement errors. To examine practical implications of the suggested methods of analysis, applications are made to GRE G eneral A nalytical W riting and TOEFL i BT W riting. Results obtained indicate that substantial improvements are possible both in terms of reliability of scoring and in terms of assessment reliability.