Premium
Delta: A new measure of agreement between two raters
Author(s) -
Andrés A. Martín,
Marzo P. Femia
Publication year - 2004
Publication title -
british journal of mathematical and statistical psychology
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 3.157
H-Index - 51
eISSN - 2044-8317
pISSN - 0007-1102
DOI - 10.1348/000711004849268
Subject(s) - kappa , measure (data warehouse) , mathematics , categorical variable , agreement , statistics , cohen's kappa , independence (probability theory) , zero (linguistics) , delta , value (mathematics) , econometrics , computer science , data mining , physics , geometry , linguistics , philosophy , astronomy
The most common measure of agreement for categorical data is the coefficient kappa. However, kappa performs poorly when the marginal distributions are very asymmetric, it is not easy to interpret, and its definition is based on hypothesis of independence of the responses (which is more restrictive than the hypothesis that kappa has a value of zero). This paper defines a new measure of agreement, delta, ‘the proportion of agreements that are not due to chance’, which comes from model of multiple‐choice tests and does not have the previous limitations. The paper shows that kappa and delta generally take very similar values, except when the marginal distributions are strongly unbalanced. The case of the 2 × 2 tables (which admits very simple solutions) is considered in detail.