Premium
Effects of Behavioral Anchors on Peer Evaluation Reliability
Author(s) -
Ohland Matthew W.,
Layton Richard A.,
Loughry Misty L.,
Yuhasz Amy G.
Publication year - 2005
Publication title -
journal of engineering education
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 3.896
H-Index - 108
eISSN - 2168-9830
pISSN - 1069-4730
DOI - 10.1002/j.2168-9830.2005.tb00856.x
Subject(s) - reliability (semiconductor) , psychology , multidisciplinary approach , inter rater reliability , applied psychology , outcome (game theory) , engineering education , medical education , mathematics education , engineering , rating scale , engineering management , developmental psychology , medicine , mathematics , social science , power (physics) , physics , mathematical economics , quantum mechanics , sociology
This paper presents comparisons of three peer evaluation instruments tested among students in undergraduate engineering classes: a single‐item instrument without behavioral anchors, a ten‐item instrument, and a single‐item behaviorally anchored instrument. Studies using the instruments in undergraduate engineering classes over four years show that the use of behavioral anchors significantly improves the inter‐rater reliability of the single‐item instrument. The inter‐rater reliability (based on four raters) of the behaviorally anchored instrument was 0.78, which was not significantly higher than that of the ten‐item instrument (0.74), but it was substantially more parsimonious. The results of this study add to the body of knowledge on evaluating students' performance in teams. This is critical since the ability to function in multidisciplinary teams is a required student learning outcome of engineering programs.