Premium
MULTIPLE ASSESSMENT OF MANAGERIAL EFFECTIVENESS: INTERRATER AGREEMENT AND CONSENSUS IN EFFECTIVENESS MODELS
Author(s) -
TSUI ANNE S.,
OHLOTT PATRICIA
Publication year - 1988
Publication title -
personnel psychology
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 6.076
H-Index - 142
eISSN - 1744-6570
pISSN - 0031-5826
DOI - 10.1111/j.1744-6570.1988.tb00654.x
Subject(s) - psychology , inter rater reliability , conceptualization , perception , preference , social psychology , agreement , empirical research , rating scale , econometrics , statistics , developmental psychology , mathematics , artificial intelligence , neuroscience , computer science , economics , linguistics , philosophy
Research has consistently identified poor interrater agreement among multiple assessments of managerial performance. Three alternative sources of dissensus in the effectiveness ratings were examined: rating errors, selective perceptions, and variations in criteria type or weight. As the available empirical evidence and theoretical analysis show, all three causes provide plausible reasons—though in varying degrees—for the low agreement coefficient. However, an empirical study designed to test three specific hypotheses on criterion type and criterion weights found consensus in the effectiveness models of superiors, subordinates, and peers. Consensus among different raters was high on both the role behaviors and on the personal traits of the managers as criteria for effectiveness. While these findings supported Biddle's role theory (1979), disagreement on the relative weights of these criteria was evident. These observations underscore the need for further conceptualization on the preference functions of raters as a primary source of the low convergent validity coefficients among multiple raters. Further research is also desirable on contextual and cognitive factors that may lead to shifts in criterion type and criterion weight, as well as on actual rating error tendencies among different raters.