Open AccessWhat is the Validity Evidence for Assessments of Clinical Teaching?
Author(s) -
Beckman Thomas J.,
Cook David A.,
Mandrekar Jayawant N.
Publication year - 2005
Publication title -
journal of general internal medicine
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.746
H-Index - 180
eISSN - 1525-1497
pISSN - 0884-8734
DOI - 10.1111/j.1525-1497.2005.0258.x
Subject(s) - inter rater reliability , medicine , reliability (semiconductor) , clinical psychology , kappa , content validity , internal validity , rating scale , psychometrics , validity , evidence based practice , external validity , evidence based medicine , psychology , applied psychology , social psychology , developmental psychology , alternative medicine , pathology , power (physics) , physics , quantum mechanics , linguistics , philosophy
Background: Although a variety of validity evidence should be utilized when evaluating assessment tools, a review of teaching assessments suggested that authors pursue a limited range of validity evidence. Objectives: To develop a method for rating validity evidence and to quantify the evidence supporting scores from existing clinical teaching assessment instruments. Design: A comprehensive search yielded 22 articles on clinical teaching assessments. Using standards outlined by the American Psychological and Education Research Associations, we developed a method for rating the 5 categories of validity evidence reported in each article. We then quantified the validity evidence by summing the ratings for each category. We also calculated weighted κ coefficients to determine interrater reliabilities for each category of validity evidence. Main Results: Content and Internal Structure evidence received the highest ratings (27 and 32, respectively, of 44 possible). Relation to Other Variables, Consequences, and Response Process received the lowest ratings (9, 2, and 2, respectively). Interrater reliability was good for Content, Internal Structure, and Relation to Other Variables (κ range 0.52 to 0.96, all P values <.01), but poor for Consequences and Response Process. Conclusions: Content and Internal Structure evidence is well represented among published assessments of clinical teaching. Evidence for Relation to Other Variables, Consequences, and Response Process receive little attention, and future research should emphasize these categories. The low interrater reliability for Response Process and Consequences likely reflects the scarcity of reported evidence. With further development, our method for rating the validity evidence should prove useful in various settings.