Errors in meta‐analyses of selection

Given the multiple commentaries on Morrissey (2016) published here, we focus our comments on Morrissey’s re-analyses of previous synthetic analyses of phenotypic selection gradients (b and c). Morrissey argues that formal meta-analyses of selection that account for sampling error are important; we agree and have conducted such analyses (Kingsolver et al., 2012). We are therefore surprised at Morrissey’s focus here on the Kingsolver et al. (2001) study and its associated nonparametric analyses, given the greatly expanded data sets, additional analyses and new statistical tools that have become available in the past 15 years (Kingsolver et al., 2001; Hereford et al., 2004; Siepielski et al., 2009, 2011; Hadfield, 2010; Kingsolver & Diamond, 2011; Morrissey & Hadfield, 2012; Nakagawa & Santos, 2012; Koricheva et al., 2013). In Kingsolver et al. (2012), we conducted mixed-effects modelling of b in a Bayesian framework using MCMCglmm (Hadfield, 2010). This framework accounted for potential effects of both sampling errors and studyand species-level autocorrelation. We also incorporated moderator variables (e.g. trait type, fitness component or taxonomic group) in the models to evaluate whether moderators influenced b. [The model statements used in these analyses were provided in the Supplement of Kingsolver et al. (2012).] We applied these analyses to the data set of Kingsolver and Diamond (2011) [updated and much larger than that of Kingsolver et al. (2001)], to estimate the posterior distribution (and mean) of b; a folded normal distribution was then used to make inferences about |b| (Hereford et al., 2004; Morrissey & Hadfield, 2012). Contrary to Morrissey’s (2016) brief summary of Kingsolver et al. (2012), we reported in that paper that accounting for sampling error substantially reduced the estimatedmeanmagnitude of selection (|b|), but that differences in average b or |b| among different trait types, fitness components or taxonomic groups were similar for analyses that do (Kingsolver et al., 2012: Fig. 1 and Table S1) or do not (Kingsolver &Diamond, 2011: Fig. 2 and Table 1; Kingsolver et al., 2001: Figs 4 and 5) formally account for sampling error. It is unclear to us whether the quantitative differences in estimates between Kingsolver et al. (2012: Table S1) and Morrissey (2016: Table 1) are due to the use of different data sets or differences in themodels employed. Morrissey (2016) also asserts that the difference between Kingsolver et al. (2012)’s and Morrissey (2016)’s estimate of the mean absolute value of selection represents a mathematical error in Kingsolver et al. (2012)’s analysis. However, this criticism is misdirected. Morrissey (2016) incorrectly interprets the difference between ‘corrected’ and ‘uncorrected’ values of the mean absolute value of selection from Kingsolver et al. (2012) (Fig. 2) as reflecting estimates from models which do and do not incorporate standard errors. In fact, the distinction between uncorrected and corrected values reflects a much broader difference between estimates from simple summary statistics of the kernel density distribution of selection coefficients versus those from a formal meta-analytical model that incorporates standard errors and random effects structure to account for study and species-level autocorrelation (see Kingsolver et al., 2012: page 1104 for a description of the comparisons made, and Fig. 2, Table S2 for results). A common problem for conducting formal meta-analyses is that estimates of standard errors (or equivalent) may not be available. For example, despite repeated calls for such information, more than half of the estimates of b or c reported in the literature do not include values for standard errors (Kingsolver et al., 2001; Siepielski et al., 2011). Morrissey (2016) is largely silent on the issue of missing data, but his re-analyses of phenotypic selection appear to only include information from data records where standard errors are available. We argue that excluding more than half of the data throws out valuable and hard-won information and has the potential to generate biases in the resulting estimates. The goal of meta-analyses is to summarize the available evidence (Gurevitch et al., 1992): informal meta-analysis of all available data may be more valuable than analyses of a fraction of the data that includes sampling error. In addition, there are many methods for imputing unknown standard errors (reviewed in Lajeunese, 2013). For example, unknown standard errors may be imputed using: the mean or median of the known standard errors (Wiebe et al., 2006); the relationship between known standard errors and sample size, with sample size being a more commonly reported metric (Ma et al., 2008); or using multiple imputation techniques (Rubin & Schenker, 1991). Whereas none of these methods perfectly solves the issue of missing standard errors, they can provide an important bridge between informal and formal meta-analysis until better reporting practices are adopted in the literature. Indeed, although imputation methods have the potential to introduce bias, comparisons of meta-analyses that use only data with available variances versus those that use data sets with imputed variances have yielded similar results (Philbrook et al., 2007). At a minimum, a comparison of results from reduced data sets Correspondence: Joel G. Kingsolver, Department of Biology, University of North Carolina, Chapel Hill, NC 27599, USA. Tel.: 919 842 6291; fax: 919 962 1625; e-mail: jgking@bio.unc.edu