Premium
On population‐based measures of agreement for binary classifications
Author(s) -
Nelson Kerrie P.,
Edwards Don
Publication year - 2008
Publication title -
canadian journal of statistics
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.804
H-Index - 51
eISSN - 1708-945X
pISSN - 0319-5724
DOI - 10.1002/cjs.5550360306
Subject(s) - cohen's kappa , kappa , statistic , statistics , population , binary number , inference , binary data , contrast (vision) , mathematics , agreement , econometrics , computer science , medicine , artificial intelligence , linguistics , geometry , environmental health , arithmetic , philosophy
The authors describe a model‐based kappa statistic for binary classifications which is interpretable in the same manner as Scott's pi and Cohen's kappa, yet does not suffer from the same flaws. They compare this statistic with the data‐driven and population‐based forms of Scott's pi in a population‐based setting where many raters and subjects are involved, and inference regarding the underlying diagnostic procedure is of interest. The authors show that Cohen's kappa and Scott's pi seriously underestimate agreement between experts classifying subjects for a rare disease; in contrast, the new statistic is robust to changes in prevalence. The performance of the three statistics is illustrated with simulations and prostate cancer data.