Premium
A Rank‐based Longitudinal Correction for Student Self‐selection When Quantifying the Effects of a Voluntary Study Reinforcement Intervention on Exam Performance
Author(s) -
Witchel Harry J.
Publication year - 2020
Publication title -
the faseb journal
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.709
H-Index - 277
eISSN - 1530-6860
pISSN - 0892-6638
DOI - 10.1096/fasebj.2020.34.s1.02869
Subject(s) - test (biology) , intervention (counseling) , cohort , psychology , medical education , psychological intervention , student engagement , mathematics education , medicine , paleontology , psychiatry , biology
The quantitative effects of voluntary online study interventions are difficult to ascribe causality to because student self‐selection can confound the analysis. An ideal solution would employ counterfactuals to link the intervention to the effect. OBJECTIVE To determine whether engaging with an online self‐testing platform improved examination performance. METHOD Institution ethical approval was received as was informed consent. A cohort of first‐year undergraduate medical students were provided with a basic online self‐test opportunity on a voluntary basis. Detailed engagement statistics were provided by the online platform, and these were compared to exam performance data over the year for exams and other assessments that normally are produced. As the same students participated in three modules, whereas only one module had the learning intervention, student’s ranks on each final examination were compared in consideration with the engagement with the learning intervention. An anonymized feedback form regarding the self‐test platform was also incorporated. RESULTS Fifty of 158 students in the cohort chose to take part and use the platform. Student feedback suggested that the self‐test platform was well liked (4.06 ± 0.10 /5) and helped them learn more (4.10 ± 0.10 /5). Students who used the platform had a trend for scoring higher on the final examination than those who did not (Cohen’s d = 0.25, unpaired t test, P < 0.10); however, there was no direct correlation between use of the intervention and examination score for the relevant module (beta = −0.0054, R‐squared = 0.029). Furthermore, in the longitudinal analysis, the comparative improvement in examination score rank for students who used the intervention (when compared to their ranks when the intervention was not available) was minimal (beta = 0.73 rank increase per 100 questions practiced, R‐squared = 0.0098). CONCLUSIONS Improved cross‐sectional performance of those students who chose to participate in the intervention is likely due to self‐selection of engaged students. The lack of clear benefits of this learning intervention when analyzed longitudinally may be due to the intervention’s focus on a small part of the curriculum that was poorly represented on the exam as well as to other known limitations in the intervention.