Programming: predicting student success early in CS1. a re-validation and replication study

This paper describes a large, multi-institutional revalidation study conducted in the academic year 2015-16. Six hundred and ninety-two students participated in this study, from 11 institutions (ten institutions in Ireland and one in Denmark). The primary goal was to validate and further develop an existing computational prediction model called Predict Student Success (PreSS). In doing so, this study addressed a call from the 2015 ITiCSE working group (the second ”Grand Challenge”), to ”systematically analyse and verify previous studies using data from multiple contexts to tease out tacit factors that contribute to previously observed outcomes”. PreSS was developed and validated in a longitudinal study conducted over a three year period (twelve years previous from 2004-06). PreSS could predict with near 80% accuracy, how a student would likely perform on an introductory programming module. Notably this could be achieved at a very early stage in the module. This paper describes a revalidation of the original PreSS model on a significantly larger multi-institutional data set twelve years after its initial development and looks at recent research on additional factors that may improve the model. The work involved the development of a fully automated end-to-end tool, which can predict student success early in CS1, with an accuracy of 71%. This paper describes, in detail the PreSS model, recent research, pilot studies and the re-validation and replication study of the PreSS model.