Direct Observation Evaluations by Emergency Medicine Faculty Do Not Provide Data That Enhance Resident Assessment When Compared to Summative Quarterly Evaluations

Objectives:  The purpose of this study was to compare quarterly global evaluations with direct observation evaluations to determine if direct observation evaluations provide unique data compared to those obtained from quarterly global evaluations. Methods:  This observational, cohort study was performed at a 3‐year emergency medicine (EM) residency program with 10 residents per year. Faculty used an online Web‐based evaluation system to complete quarterly global evaluations and patient‐specific direct observation evaluations. Two scores were collected for each resident within each quarterly evaluation period: 1) the quarterly evaluation score was the mean score across all faculty who performed a quarterly evaluation and, 2) the direct observation score was the mean score across all faculty who performed a direct observation evaluation. Pearson correlation coefficients were performed across these two groups of evaluations. Results:  Over the 4‐year period of the study 296 complete data sets were available for the analysis. When the quarterly evaluation score was correlated with the direct observation score for each resident at the same evaluation period, we found a very high correlation for each of the eight evaluation questions ( r  = 0.95–0.96, p < 0.0001). When these evaluations were stratified based on the number of direct observation evaluations that were performed during the evaluation period of interest, the correlation between the quarterly evaluation and the direct observation scores increased as the number of direct observations in the evaluation period increased. The evaluation scores from the faculty who had performed both direct observation and quarterly evaluation methods during the same resident evaluation period were highly correlated even with small numbers of evaluators. Conclusions:  Direct observations are highly correlated with quarterly evaluations when there are greater than three direct observation evaluations completed; however, this correlation drops significantly when the number of direct observations is lower. Direct observation evaluations provide similar data when compared with data obtained from quarterly global evaluations. ACADEMIC EMERGENCY MEDICINE 2010; 17:S72–S77 © 2010 by the Society for Academic Emergency Medicine