Current approaches using genetic distances produce poor estimates of landscape resistance to interindividual dispersal

Abstract Landscape resistance reflects how difficult it is for genes to move across an area with particular attributes (e.g. land cover, slope). An increasingly popular approach to estimate resistance uses M antel and partial M antel tests or causal modelling to relate observed genetic distances to effective distances under alternative sets of resistance parameters. Relatively few alternative sets of resistance parameters are tested, leading to relatively poor coverage of the parameter space. Although this approach does not explicitly model key stochastic processes of gene flow, including mating, dispersal, drift and inheritance, bias and precision of the resulting resistance parameters have not been assessed. We formally describe the most commonly used model as a set of equations and provide a formal approach for estimating resistance parameters. Our optimization finds the maximum M antel r when an optimum exists and identifies the same resistance values as current approaches when the alternatives evaluated are near the optimum. Unfortunately, even where an optimum existed, estimates from the most commonly used model were imprecise and were typically much smaller than the simulated true resistance to dispersal. Causal modelling using M antel significance tests also typically failed to support the true resistance to dispersal values. For a large range of scenarios, current approaches using a simple correlational model between genetic and effective distances do not yield accurate estimates of resistance to dispersal. We suggest that analysts consider the processes important to gene flow for their study species, model those processes explicitly and evaluate the quality of estimates resulting from their model.