Premium
Reliability, validity and efficiency of multiple choice question and patient management problem item formats in assessment of clinical competence
Author(s) -
NORCINI J. J.,
SWANSON D. B.,
GROSSO L. J.,
WEBSTER G. D.
Publication year - 1985
Publication title -
medical education
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.776
H-Index - 138
eISSN - 1365-2923
pISSN - 0308-0110
DOI - 10.1111/j.1365-2923.1985.tb01314.x
Subject(s) - competence (human resources) , multiple choice , psychology , face validity , reliability (semiconductor) , computer science , psychometrics , medical education , medicine , clinical psychology , statistics , social psychology , mathematics , significant difference , power (physics) , physics , quantum mechanics
Summary. Despite a lack of face validity, there continues to be heavy reliance on objective paper‐and‐pencil measures of clinical competence. Among these measures, the most common item formats are patient management problems (PMPs) and three types of multiple choice questions (MCQs): one‐best‐answer (A‐types); matching questions (M‐types); and multiple true/false questions (X‐types). The purpose of this study is to compare the reliability, validity and efficiency of these item formats with particular focus on whether MCQs and PMPs measure different aspects of clinical competence. Analyses revealed reliabilities of 0.72 or better for all item formats; the MCQ formats were most reliable. Similarly, efficiency analyses (reliability per unit of testing time) demonstrated the superiority of MCQs. Evidence for validity obtained through correlations of both programme directors’ ratings and criterion group membership with item format scores also favoured MCQs. More important, however, is whether MCQs and PMPs measure the same or different aspects of clinical competence. Regression analyses of the scores on the validity measures (programme directors’ ratings and criterion group membership) indicated that MCQs and PMPs seem to be measuring predominantly the same thing. MCQs contribute a small unique variance component over and above PMPs, while PMPs make the smallest unique contribution. As a whole, these results indicate that MCQs are more efficient, reliable and valid than PMPs.