Premium
Assessing transferability of ecological models: an underappreciated aspect of statistical validation
Author(s) -
Wenger Seth J.,
Olden Julian D.
Publication year - 2012
Publication title -
methods in ecology and evolution
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 3.425
H-Index - 105
ISSN - 2041-210X
DOI - 10.1111/j.2041-210x.2011.00170.x
Subject(s) - computer science , transferability , salmo , machine learning , set (abstract data type) , generality , range (aeronautics) , sample (material) , selection (genetic algorithm) , model selection , data set , data mining , sample size determination , artificial intelligence , statistics , mathematics , fishery , engineering , fish <actinopterygii> , psychology , chemistry , logit , chromatography , psychotherapist , biology , programming language , aerospace engineering
Summary 1. Ecologists have long sought to distinguish relationships that are general from those that are idiosyncratic to a narrow range of conditions. Conventional methods of model validation and selection assess in‐ or out‐of‐sample prediction accuracy but do not assess model generality or transferability, which can lead to overestimates of performance when predicting in other locations, time periods or data sets. 2. We propose an intuitive method for evaluating transferability based on techniques currently in use in the area of species distribution modelling. The method involves cross‐validation in which data are assigned non‐randomly to groups that are spatially, temporally or otherwise distinct, thus using heterogeneity in the data set as a surrogate for heterogeneity among data sets. 3. We illustrate the method by applying it to distribution modelling of brook trout ( Salvelinus fontinalis Mitchill) and brown trout ( Salmo trutta Linnaeus) in western United States. We show that machine‐learning techniques such as random forests and artificial neural networks can produce models with excellent in‐sample performance but poor transferability, unless complexity is constrained. In our example, traditional linear models have greater transferability. 4. We recommend the use of a transferability assessment whenever there is interest in making inferences beyond the data set used for model fitting. Such an assessment can be used both for validation and for model selection and provides important information beyond what can be learned from conventional validation and selection techniques.