Premium
Measurement of inter‐rater agreement for transient events using Monte Carlo sampled permutations
Author(s) -
Norman Robert G.,
Scott Marc A.
Publication year - 2006
Publication title -
statistics in medicine
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.996
H-Index - 183
eISSN - 1097-0258
pISSN - 0277-6715
DOI - 10.1002/sim.2568
Subject(s) - monte carlo method , confidence interval , transient (computer programming) , event (particle physics) , permutation (music) , statistics , computer science , inter rater reliability , polysomnography , statistical physics , algorithm , mathematics , physics , medicine , apnea , rating scale , astrophysics , acoustics , operating system
In this paper we demonstrate the adverse effect of serially observed data sequences containing transient events on the calculation of Cohen's κ as an index of inter‐rater agreement in the detection of these events. We develop and use a Monte‐Carlo‐based permutation technique to produce an empiric distribution of κ in the presence of serial dependence. We find that the empiric confidence intervals for κ tend to be wider than parametrically derived intervals and in the case of longer event lengths, are markedly so. We evaluate the effect of number and length of events, and further, describe and evaluate three permutation methods which match specific rating situations. Finally, we apply these techniques to the measurement of inter‐rater agreement for sleep disordered breathing events, a transient event identified during nocturnal polysomnography, for which traditionally computed confidence intervals for κ are incorrect. Copyright © 2006 John Wiley & Sons, Ltd.