Premium
Comparing holistic and disaggregated ratings in the evaluation of scientific presentations
Author(s) -
Arkes Hal R.,
Shaffer Victoria A.,
Dawes Robyn M.
Publication year - 2006
Publication title -
journal of behavioral decision making
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.136
H-Index - 76
eISSN - 1099-0771
pISSN - 0894-3257
DOI - 10.1002/bdm.503
Subject(s) - psychology , econometrics , cognitive psychology , economics
The National Institutes of Health refused to switch to disaggregated ratings as a method for evaluating proposals, because no contest between disaggregated and holistic ratings had ever used scientific materials as the to‐be‐rated stimuli. We designed two studies to fill this research void. Participants rated scientific convention presentations either using a holistic procedure in which one overall rating was given or a disaggregated procedure in which one rating was given to each of five criteria. In four of the six convention sessions the disaggregated ratings led to higher inter‐rater reliability than did the holistic ratings; three of these differences were statistically significant. The inter‐rater reliabilities between the two types of ratings collapsed across all sessions differed significantly. In a second experiment, participants rated posters using either disaggregated or holistic ratings. The disaggregated ratings again led to higher inter‐rater reliability, but not significantly so. In 35 of the 43 sessions in which disaggregated and holistic ratings were compared, the variance of the disaggregated ratings was smaller. Copyright © 2006 John Wiley & Sons, Ltd.