Premium
Measuring interrater reliability among multiple raters: An example of methods for nominal data
Author(s) -
Posner Karen L.,
Sampson Paul D.,
Caplan Robert A.,
Ward Richard J.,
Cheney Frederick W.
Publication year - 1990
Publication title -
statistics in medicine
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.996
H-Index - 183
eISSN - 1097-0258
pISSN - 0277-6715
DOI - 10.1002/sim.4780090917
Subject(s) - jackknife resampling , inter rater reliability , reliability (semiconductor) , estimator , statistics , computer science , kappa , scale (ratio) , standard error , data set , set (abstract data type) , observational error , econometrics , mathematics , rating scale , power (physics) , physics , geometry , quantum mechanics , programming language
This paper reviews and critiques various approaches to the measurement of reliability among multiple raters in the case of nominal data. We consider measurement of the overall reliability of a group of raters (using kappa‐like statistics) as well as the reliability of individual raters with respect to a group. We introduce modifications of previously published estimators appropriate for measurement of reliability in the case of stratified sampling frames and we interpret these measures in view of standard errors computed using the jackknife. Analyses of a set of 48 anaesthesia case histories in which 42 anaesthesiologists independently rated the appropriateness of care on a nominal scale serve as an example.