A Monte-Carlo analysis of competitive balance and reliability across tournament structures

This paper investigates the effect of increasing competitive balance on the reliability of tournament rankings. Reliability of rankings, a previously qualitative property, is quantified in this paper by the closeness between ground truth rankings and the rankings of teams at the end of a tournament. Three metrics are used to measure this closeness: Spearman’s rank correlation coefficient, Kendall’s tau, and a relatively unused algorithm in the field of ranking: Levenshtein distance. Three tournament structures are simulated: round-robin, random pairings, and the Swiss system. The tournaments are simulated across multiple trials and over a varying number of games. It is found that the rate of growth of reliability of a tournament structure falls as the number of games increases. It is also found that there is a positive relationship between competitive imbalance and reliability. The marginal benefit of increasing competitive imbalance falls as it is increased. Unexpectedly, in comparison to random pairings and Swiss pairings, the round-robin tournament structure is seen to achieve the highest reliability score across all metrics and number of games played. The difference in reliability between the tournament structures increases as competitive imbalance is increased. The further work suggested includes investigation of tournament outcome uncertainty in conjunction with reliability and competitive balance, a closer study into Levenshtein distance as a useful algorithm to quantify closeness between two rankings, and an inquiry into the specific factors that bottleneck reliability while the number of games played in a tournament increases.