Open AccessCovariance of pairwise differences on a multi-species coalescent tree and implications forFST
Author(s) -
Geno Guerra,
Rasmus Nielsen
Publication year - 2022
Publication title -
philosophical transactions - royal society. biological sciences
Language(s) - English
Resource type - Journals
eISSN - 1471-2970
pISSN - 0962-8436
DOI - 10.1098/rstb.2020.0415
Subject(s) - coalescent theory , pairwise comparison , covariance , estimator , mathematics , population , phylogenetic tree , piecewise , effective population size , statistics , coalescence (physics) , biology , evolutionary biology , genetic variation , genetics , gene , mathematical analysis , demography , sociology , astrobiology
The multi-species coalescent (MSC) provides a theoretical foundation for modern phylogenetics and comparative population genetics. Its theoretical properties have been heavily studied but there are still aspects of the MSC that are largely unknown, including the covariances in pairwise coalescence times, which are fundamental for understanding the properties of statistics that combine data from multiple species, such as the fixation index (F ST ). The major contribution of this study is the derivation and implementation of exact expressions for the covariances of pairwise coalescence times under phylogenetic models with piecewise constant changes in population size, assuming no gene flow after species divergence. We use these expressions to derive the variance in average pairwise differences within and between populations. We then derive approximations for the expectation and bias of a sequence-based estimator ofF ST , a commonly used genetic measurement of population differentiation, when it is applied to a non-recombining region of the genome. We show that the estimator ofF ST is generally biased downward. A freely available software package is provided, STCov, to calculate the mean, variances and covariances in coalescence times presented here under user-defined piecewise-constant species trees.This article is part of the theme issue ‘Celebrating 50 years since Lewontin's apportionment of human diversity’.