Premium
Rater Reliability: Short‐ and Long‐Term Effects of Calibration Training
Author(s) -
HajAli Reem,
Feil Philip
Publication year - 2006
Publication title -
journal of dental education
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.53
H-Index - 68
eISSN - 1930-7837
pISSN - 0022-0337
DOI - 10.1002/j.0022-0337.2006.70.4.tb04097.x
Subject(s) - gold standard (test) , grading (engineering) , rating scale , inter rater reliability , calibration , test (biology) , grading scale , psychology , medicine , term (time) , amalgam (chemistry) , physical therapy , statistics , mathematics , surgery , paleontology , civil engineering , electrode , chemistry , quantum mechanics , biology , physics , engineering
The purpose of this investigation was to evaluate the immediate effects of calibration on inter‐rater agreement to a gold standard (GS) and to determine whether the effects can be sustained over a ten‐week period. Valid criteria for a Class II amalgam preparation, a three‐point rating scale, and a grade form were developed. Three tests were administered: prior to calibration training, immediately following training, and ten weeks later. Each test consisted of faculty independently evaluating ten prepared teeth. Agreement with GS scores for most of the grading criteria improved as a result of training and did not deteriorate over time. The overall percent agreement was 54.5, 66.9, and 64.6 percent across test periods. The most impressive gains in agreement occurred when the criteria evaluated had a GS score of either “standard not met” or “ideal.” There was very little gain when the gold standard score was “acceptable.” It is concluded that, with training, inter‐rater agreement with a gold standard can improve and such improvement is reasonably resistant to deterioration after ten weeks. Nevertheless, future training ought to consider the use of a mastery approach in calibration training to ensure that a satisfactory degree of agreement with the GS is obtained.