Premium
Reliability of recording instructions: Multivariate agreement for nominal data
Author(s) -
Krippendorff Klaus
Publication year - 1971
Publication title -
behavioral science
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.371
H-Index - 45
eISSN - 1099-1743
pISSN - 0005-7940
DOI - 10.1002/bs.3830160305
Subject(s) - reliability (semiconductor) , computer science , relation (database) , process (computing) , reliability engineering , quality (philosophy) , term (time) , variable (mathematics) , data mining , power (physics) , scheme (mathematics) , mathematics , engineering , mathematical analysis , philosophy , physics , epistemology , quantum mechanics , operating system
A method is proposed for evaluating the reliability of the details of such recording instructions as may be given to coders in content analysis or to observers of social interaction. The method is applicable to situations in which (1) the data to be generated are nominal, (2) the number of coders or observers are variable during the process, and (3) the instructions specify a multilevel decision process. The chief aim of the analysis is to detect and evaluate sources of unreliability in complex recording instructions and to predict by computational means the reliability of data when parts of a recording instruction are deleted. In contrast to other aims of data quality control, the term instruction reliability is used to designate the results. The method provides several coefficients and an accounting scheme for amount of disagreement. The power of the method is demonstrated by a computational example, and the relation of the proposed measures to other known reliability coefficients is discussed.