Open AccessLarge-Sample Variance of Fleiss Generalized Kappa
Author(s) -
Kilem L. Gwet
Publication year - 2021
Publication title -
educational and psychological measurement
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.819
H-Index - 95
eISSN - 1552-3888
pISSN - 0013-1644
DOI - 10.1177/0013164420973080
Subject(s) - kappa , variance (accounting) , statistics , mathematics , sample (material) , cohen's kappa , monte carlo method , statistical inference , sample variance , chemistry , geometry , accounting , chromatography , business
Cohen's kappa coefficient was originally proposed for two raters only, and it later extended to an arbitrarily large number of raters to become what is known as Fleiss' generalized kappa. Fleiss' generalized kappa and its large-sample variance are still widely used by researchers and were implemented in several software packages, including, among others, SPSS and the R package "rel." The purpose of this article is to show that the large-sample variance of Fleiss' generalized kappa is systematically being misused, is invalid as a precision measure for kappa, and cannot be used for constructing confidence intervals. A general-purpose variance expression is proposed, which can be used in any statistical inference procedure. A Monte-Carlo experiment is presented, showing the validity of the new variance estimation procedure.