Premium
The reliability of summative judgements based on objective structured clinical examination cases distributed across the clinical year
Author(s) -
Bergus George R,
Kreiter Clarence D
Publication year - 2007
Publication title -
medical education
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.776
H-Index - 138
eISSN - 1365-2923
pISSN - 0308-0110
DOI - 10.1111/j.1365-2923.2007.02786.x
Subject(s) - summative assessment , objective structured clinical examination , reliability (semiconductor) , psychology , medical education , medicine , medical physics , formative assessment , mathematics education , power (physics) , physics , quantum mechanics
Context Objective structured clinical examinations (OSCEs) can be used for formative and summative evaluation. We sought to determine the generalisability of students' summary scores aggregated from formative OSCE cases distributed across 5 clerkships during Year 3 of medical school. Methods Five major clerkships held OSCEs with 2−4 cases each during their rotations. All cases used 15‐minute student−standardised patient encounters and performance was assessed using clinical and communication skills checklists. As not all students completed every clerkship or OSCE case, the generalisability (G) study was an unbalanced student × (case : clerkship) design. After completion of the G study, a decision (D) study was undertaken and phi (φ) values for different cut‐points were calculated. Results The data for this report were collected over 2 academic years involving 262 Year 3 students. The G study found that 9.7% of the score variance originated from the student, 3.1% from the student–clerkship interaction, and 87.2% from the student−case nested within clerkship effect. Using the variance components from the G study, the D study suggested that if students completed 3 OSCE cases in each of the 5 different clerkships, the reliability of the aggregated scores would be 0.63. The φ, calculated at a cut‐point 1 standard deviation below the mean, would be approximately 0.85. Conclusions Aggregating case scores from low stakes OSCEs within clerkships results in a score set that allows for very reliable decisions about which students are performing poorly. Medical schools can use OSCE case scores collected over a clinical year for summative evaluation.