Premium
Very‐short‐answer questions: reliability, discrimination and acceptability
Author(s) -
Sam Amir H,
Field Samantha M,
Collares Carlos F,
Vleuten Cees P M,
Wass Val J,
Melville Colin,
Harris Joanne,
Meeran Karim
Publication year - 2018
Publication title -
medical education
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.776
H-Index - 138
eISSN - 1365-2923
pISSN - 0308-0110
DOI - 10.1111/medu.13504
Subject(s) - reliability (semiconductor) , context (archaeology) , test (biology) , psychology , audiology , medicine , paleontology , power (physics) , physics , quantum mechanics , biology
Context Single‐best‐answer questions ( SBAQ s) have been widely used to test knowledge because they are easy to mark and demonstrate high reliability. However, SBAQ s have been criticised for being subject to cueing. Objectives We used a novel assessment tool that facilitates efficient marking of open‐ended very‐short‐answer questions ( VSAQ s). We compared VSAQ s with SBAQ s with regard to reliability, discrimination and student performance, and evaluated the acceptability of VSAQ s. Methods Medical students were randomised to sit a 60‐question assessment administered in either VSAQ and then SBAQ format (Group 1, n = 155) or the reverse (Group 2, n = 144). The VSAQ s were delivered on a tablet; responses were computer‐marked and subsequently reviewed by two examiners. The standard error of measurement ( SEM ) across the ability spectrum was estimated using item response theory. Results The review of machine‐marked questions took an average of 1 minute, 36 seconds per question for all students. The VSAQ s had high reliability (alpha: 0.91), a significantly lower SEM than the SBAQ s (p < 0.001) and higher mean item–total point biserial correlations (p < 0.001). The VSAQ scores were significantly lower than the SBAQ scores (p < 0.001). The difference in scores between VSAQ s and SBAQ s was attenuated in Group 2. Although 80.4% of students found the VSAQ s more difficult, 69.2% found them more authentic. Conclusions The VSAQ format demonstrated high reliability and discrimination and items were perceived as more authentic. The SBAQ format was associated with significant cueing. The present results suggest the VSAQ format has a higher degree of validity.