Premium
An implicit enumeration method for an exact test of weighted kappa
Author(s) -
Brusco Michael J.,
Stahl Stephanie,
Steinley Douglas
Publication year - 2008
Publication title -
british journal of mathematical and statistical psychology
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 3.157
H-Index - 51
eISSN - 2044-8317
pISSN - 0007-1102
DOI - 10.1348/000711007x227058
Subject(s) - kappa , enumeration , mathematics , resampling , statistics , contingency table , value (mathematics) , cohen's kappa , measure (data warehouse) , table (database) , combinatorics , discrete mathematics , algorithm , computer science , data mining , geometry
The kappa coefficient is one of the most widely used measures for evaluating the agreement between two raters asked to assign N objects to one of K nominal categories. Weighted versions of kappa enable partial credit to be awarded for near agreement, most notably in the case of ordinal categories. An exact significance test for weighted kappa can be conducted by enumerating all rater agreement tables with the same fixed marginal frequencies as the observed table, and accumulating the probabilities for all tables that produce a weighted kappa index that is greater than or equal to the observed measure. Unfortunately, complete enumeration of all tables is computationally unwieldy for modest values of N and K . We present an implicit enumeration algorithm for conducting an exact test of weighted kappa, which can be applied to tables of non‐trivial size. The algorithm is particularly efficient for ‘good’ to ‘excellent’ values of weighted kappa that typically have very small p ‐values. Therefore, our method is beneficial for situations where resampling tests are of limited value because the number of trials needed to estimate the p ‐value tends to be large.