Premium
MEASURING AGREEMENT WHEN TWO OBSERVERS CLASSIFY PEOPLE INTO CATEGORIES NOT DEFINED IN ADVANCE
Author(s) -
Brennan Robert L.,
Light Richard J.
Publication year - 1974
Publication title -
british journal of mathematical and statistical psychology
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 3.157
H-Index - 51
eISSN - 2044-8317
pISSN - 0007-1102
DOI - 10.1111/j.2044-8317.1974.tb00535.x
Subject(s) - statistic , mathematics , categorization , statistics , sample (material) , contrast (vision) , test statistic , variance (accounting) , measure (data warehouse) , set (abstract data type) , null hypothesis , agreement , scale (ratio) , level of measurement , null distribution , distribution (mathematics)
Basic to many psychological investigations is the question of agreement between observers who independently categorize people. Several recent studies have proposed measures of agreement when a set of nominal scale categories has been predefined and imposed on two observers. This study, in contrast, develops a measure of agreement for settings where observers independently define their own categories. Thus it is possible for observers to delineate different numbers of categories, with different names. Computational formulae for the mean and variance of the proposed measure of agreement are given; further, a statistic with a large‐sample normal distribution is suggested for testing the null hypothesis of random agreement. A computer‐based comparison of the large‐sample approximation with the exact distribution of the test statistic shows a generally good fit, even for moderate sample sizes. Finally, a worked example involving two psychologists' classifications of children illustrates the computations.