Ratings and Rankings: Multiple Comparisons of Mean Ratings (Research Note)

Ranking of journals or other objects according to mean ratings computed from an opinion survey is shown to be inappropriate if a test of significance shows no difference between them. A Scheffe test for comparisons of mean ratings of journals ranked by Kohl and Davis [C&RL 46:40-47 (Jan. 1985)] was performed. The results indicate no significant difference between means. Confidence intervals for every adjacent pair of journals in the list of ratings by ARL directors were also computed. The results indicate that every adjacent interval overlaps, and that the means are essentially tie scores. Treating them as significantly different, therefore, is a Type 1 error.