Premium
Validity evidence of resident competency ratings and the identification of problem residents
Author(s) -
Park Yoon Soo,
Riddle Janet,
Tekian Ara
Publication year - 2014
Publication title -
medical education
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.776
H-Index - 138
eISSN - 1365-2923
pISSN - 0308-0110
DOI - 10.1111/medu.12408
Subject(s) - identification (biology) , medical education , psychology , medline , educational measurement , family medicine , medicine , curriculum , pedagogy , political science , botany , law , biology
Objectives This study examined validity evidence of end‐of‐rotation evaluations used to measure progress toward mastery of core competencies in residents. In addition, this study investigated whether end‐of‐rotation evaluations can be used to detect problem residents during their training. Methods Historical data for a 4‐year period (2009–2012), containing 4986 observations of 291 internal medicine residents, were examined. Residents were observed and assessed by fellows, faculty members and programme directors on nine domains, including the six Accreditation Council for Graduate Medical Education core competencies, as part of their end‐of‐rotation evaluations. Descriptive statistics were used to collect evidence of the response process. Correlations between competencies and a generalisability study were used to examine the internal structure of the end‐of‐rotation evaluations. Hierarchical regression was used to estimate the increase in scores across years of training. Scores on end‐of‐rotation evaluations were compared with trainees identified as problem residents by programme directors. Results Compared with fellows, faculty and programme directors had significantly greater variability in assigning scores across different competencies. Correlations between competencies ranged from 0.69 to 0.92. The reliability of end‐of‐rotation evaluations was adequate (fellows, phi coefficient [φ] = 0.68; faculty [including programme directors], φ = 0.71). Mean scores increased by 0.21 points (95% confidence interval 0.18–0.24) per postgraduate year. Mean scores were significantly correlated with classification as a problem resident ( r = 0.33, p < 0.001); problem residents also had significantly lower ratings across all competencies during PGY‐1 compared with all other residents. Conclusions End‐of‐rotation evaluations are a useful method of measuring the growth in resident performance associated with core competencies when sufficient numbers of end‐of‐rotation evaluation scores are used. Furthermore, end‐of‐rotation evaluation scores provide preliminary evidence with which to detect and predict problem residents in subsequent postgraduate training years.