Detecting a Trend in Paired Rankings

SUMMARY Each subject in a study independently ranks a set of items, undergoes a treatment and afterwards ranks the same set of items. Two tests are proposed of the null hypothesis that the conditional distribution of each post‐treatment ranking is symmetrically distributed about the corresponding preranking. The alternative hypothesis for each test is that the post‐treatment rankings are more clustered about a particular idealized ranking than would be expected under the null hypothesis. The idea of the first test is to score each subject according to how close his post‐ranking is to the idealized ranking, conditional on the distance from his post‐ranking to his preranking. The second test controls for a possible additional trend towards a potentially confounding ranking, such as the rank order in which the items are presented. The methodology is applied to a data set where 38 students rank four styles of literary criticism, then undergo a course in literary appreciation and afterwards rank the same four styles of criticism. A brief power study is also presented.