Retest effects in medical selection interviews

Context Repetition of a cognitive ability test is known to increase scores, but almost no research has examined whether similar improvement occurs with repetition of interviews. Retest effects can change the rank order of candidates and reduce the test's criterion validity. Because interviews are widely used to select medical students and postgraduate trainees, and because applicants apply to multiple programmes and often reapply if unsuccessful, the potential for retest effects needs to be understood. Objectives This study was designed to identify if retest improvements occur when candidates undertake multiple interviews and, if so, whether the effect is attributable to general interview experience or specific experience and whether repeat testing affects criterion validity. Methods We compared interview scores of applicants who were interviewed for one or more of three independent undergraduate medical programmes in two consecutive years and those who were interviewed in both years for the same programme. Correlations between initial and repeat interview scores and a written test of social understanding were compared. Results General experience (being interviewed by multiple programmes) did not produce improvement in subsequent interview performance. There was no evidence of method effect (having prior experience of the multiple mini‐interview process). Specific experience (being interviewed by the same programme across 2 years) resulted in a significant improvement in scores for which regression to the mean did not fully account. Criterion validity did not appear to be affected. Conclusions Unsuccessful candidates for medical school who reapply and are re‐interviewed on a subsequent occasion at the same institution are likely to increase their scores. The results of this study suggest the increase is probably not attributable to improved ability.