Premium
Overall indices for assessing agreement among multiple raters
Author(s) -
Jang Jeong Hoon,
Manatunga Amita K.,
Taylor Andrew T.,
Long Qi
Publication year - 2018
Publication title -
statistics in medicine
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.996
H-Index - 183
eISSN - 1097-0258
pISSN - 0277-6715
DOI - 10.1002/sim.7912
Subject(s) - pairwise comparison , statistics , mathematics , homogeneity (statistics) , inference , reliability (semiconductor) , set (abstract data type) , context (archaeology) , mean squared error , standard deviation , index (typography) , computer science , econometrics , artificial intelligence , paleontology , power (physics) , physics , quantum mechanics , world wide web , biology , programming language
The need to assess agreement exists in various clinical studies where quantifying inter‐rater reliability is of great importance. Use of unscaled agreement indices, such as total deviation index and coverage probability (CP), is recommended for two main reasons: (i) they are intuitive in a sense that interpretations are tied to the original measurement unit; (ii) practitioners can readily determine whether the agreement is satisfactory by directly comparing the value of the index to a prespecified tolerable CP or absolute difference. However, the unscaled indices were only defined in the context of comparing two raters or multiple raters that assume homogeneity of variances across raters. In this paper, we introduce a set of overall indices based on the root mean square of pairwise differences that are unscaled and can be used to evaluate agreement among multiple raters that often exhibit heterogeneous measurement processes in practice. Furthermore, we propose another overall agreement index based on the root mean square of pairwise differences that is scaled and extends the concept of the recently proposed relative area under CP curve in the presence of multiple raters. We present the definitions of overall indices and propose inference procedures in which bootstrap methods are used for the estimation of standard errors. We assess the performance of the proposed approach and demonstrate its superiority over the existing methods when raters exhibit heterogeneous measurement processes using simulation studies. Finally, we demonstrate the application of our methods using a renal study.