Premium
8.5.2 Evaluation of Interrater Agreement in Internal Audit based on ISO 9001:2000
Author(s) -
Hwang Young Ha,
Yi Changoo
Publication year - 2004
Publication title -
incose international symposium
Language(s) - English
Resource type - Journals
ISSN - 2334-5837
DOI - 10.1002/j.2334-5837.2004.tb00611.x
Subject(s) - inter rater reliability , kappa , audit , reliability (semiconductor) , cohen's kappa , quality (philosophy) , quality audit , statistics , accounting , operations management , business , mathematics , engineering , physics , power (physics) , rating scale , geometry , quantum mechanics
An internal audit in quality management system based on ISO 9001:2000 is an important process that continually improves the effectiveness and efficiency of a quality management system. The reliability of assessments of an internal audit based on ISO 9001:2000 is an issue that continues to demand the attention of researchers in this field. One element of reliability is the extent to which different teams assessing the same project produce similar ratings when presented with the same evidence. Cohen's Kappa coefficient has been the most popular measure of reliability for measuring the degree of interrater agreement beyond chance. However, the application of the Kappa coefficient may be biased downwards due to two Kappa “paradoxes.” To compensate for the deficiency of the Kappa coefficient, we additionally use a simpler index of observed agreement between two teams of auditors. Using Cohen's Kappa coefficient and the observed agreement, this paper presents some results from two assessments between two teams conducted during internal audits in Electronics and Telecommunications Research Institute in Korea. In each of these assessments, two independent teams performed the same ratings. The results indicate that in general there is at least moderate agreement between the two teams and the extent of agreement between the two teams is substantial.