Premium
Reducing the number of options on multiple‐choice questions: response time, psychometrics and standard setting
Author(s) -
Schneid Stephen D,
Armour Chris,
Park Yoon Soo,
Yudkowsky Rachel,
Bordage Georges
Publication year - 2014
Publication title -
medical education
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.776
H-Index - 138
eISSN - 1365-2923
pISSN - 0308-0110
DOI - 10.1111/medu.12525
Subject(s) - multiple choice , pharmacy , reliability (semiconductor) , educational measurement , united states medical licensing examination , medicine , item analysis , psychometrics , health professions , construct (python library) , test (biology) , construct validity , psychology , medical education , clinical psychology , family medicine , medical school , significant difference , computer science , curriculum , health care , pedagogy , economic growth , biology , paleontology , power (physics) , quantum mechanics , programming language , physics , economics
Objectives Despite significant evidence supporting the use of three‐option multiple‐choice questions ( MCQ s), these are rarely used in written examinations for health professions students. The purpose of this study was to examine the effects of reducing four‐ and five‐option MCQ s to three‐option MCQ s on response times, psychometric characteristics, and absolute standard setting judgements in a pharmacology examination administered to health professions students. Methods We administered two versions of a computerised examination containing 98 MCQ s to 38 Year 2 medical students and 39 Year 3 pharmacy students. Four‐ and five‐option MCQ s were converted into three‐option MCQ s to create two versions of the examination. Differences in response time, item difficulty and discrimination, and reliability were evaluated. Medical and pharmacy faculty judges provided three‐level Angoff ( TLA ) ratings for all MCQ s for both versions of the examination to allow the assessment of differences in cut scores. Results Students answered three‐option MCQ s an average of 5 seconds faster than they answered four‐ and five‐option MCQ s (36 seconds versus 41 seconds; p = 0.008). There were no significant differences in item difficulty and discrimination, or test reliability. Overall, the cut scores generated for three‐option MCQ s using the TLA ratings were 8 percentage points higher (p = 0.04). Conclusions The use of three‐option MCQ s in a health professions examination resulted in a time saving equivalent to the completion of 16% more MCQ s per 1‐hour testing period, which may increase content validity and test score reliability, and minimise construct under‐representation. The higher cut scores may result in higher failure rates if an absolute standard setting method, such as the TLA method, is used. The results from this study provide a cautious indication to health professions educators that using three‐option MCQ s does not threaten validity and may strengthen it by allowing additional MCQ s to be tested in a fixed amount of testing time with no deleterious effect on the reliability of the test scores.