Anomalous Unrooted Gene Trees

The coalescent and multispecies coalescent model rooted genealogies backward through time. Often, the direction of time is unknown in trees estimated from molecular sequences due to reversible mutation models, absence of an appropriate outgroup, and the absence of the molecular clock. In this article, probabilities of unrooted gene-tree topologies under the multispecies coalescent are considered. The main result is that for any species-tree topology with seven or more taxa, there exist branch lengths such that there are unrooted gene-tree topologies that are more likely than the one that has the same unrooted topology as the species tree. Species trees with such anomalous unrooted gene trees (AUGTs) are characterized for trees with five and six taxa, and patterns of branch lengths leading to AUGTs and rooted anomalous gene trees are explored. The results could be useful for understanding gene tree discordance and designing simulations studies for inferring challenging species trees.