Discussion (3): Jones–Johnson Paper

We commend Jones and Johnson for providing a clear and concise introduction to the quickly expanding field of statistical design and analysis of computer experiments. Our own experiences confirm that computer models are widely used in many areas of science and engineering, and the need to understand the performance of these models is critical. This is particularly true in light of the fact that with improving model fidelity, they are increasingly used to certify complex engineering systems with decreasing reliance on expensive physical experiments. As the authors indicate, many computer models are sufficiently complex that only a small budget of model runs is allowed for any given application. Therefore, concepts of statistical experiment design become relevant for the purpose of intelligently selecting runs to inform the development of a statistical surrogate for model output—often referred to as an emulator—that will serve as the basis for statistical inference. There are several practical issues with emulating any computer model. Is the standard Gaussian Process (GP) model a good choice for general applications? What mean and covariance structure should one choose? How should the budget of runs be expended? The authors addressed these issues effectively in their article. We provide some additional perspective in what follows. Extensive literature (see Sacks et al.1, Santner et al.2) and experience suggest that the GP model is an ideal candidate for building an emulator. Ben-Ari and Steinberg3 conducted an extensive simulation study comparing the GP model with a large class of competing models and found that GP-based emulation performs well in many situations. There are several practical considerations that must be addressed when using the GP model. Arguably the most important is how many runs are needed to adequately emulate the computer model. Loeppky et al.4 argue that the often quoted rule of ‘n=10d’ (i.e. 10 model runs per dimension) generally provides sufficient information for emulation. The design chosen for the example of this article comes close to attaining this target, at 8.75d . In addition to run size considerations, it is important to calculate diagnostics that directly assess the quality of model fit. The root mean square error (RMSE) is an obvious criterion; however, holdout samples are often unavailable in practice. In such cases the cross-validated (CV)-RMSE (see Welch et al.5) and the individual CV residuals are useful. The remainder of this discussion is structured to draw attention to additional connections between traditional response surface methodology (RSM) and analysis of computer experiments using the GP model. In particular, we focus on two basic components of response surface methods (see Box and Wilson6) for which recent developments have made analogues available for analysis of computer experiments: sensitivity analysis and sequential optimization. Sensitivity analysis refers to measuring the impact of input variations on output uncertainty. In particular, output uncertainty can be decomposed into main and interaction effects analogous to traditional analysis of variance (ANOVA), and sensitivity indices measuring the contribution of these individual effects to the total output variance can be computed (see Saltelli et al.7, Oakley and O’Hagan8, Schonlau and Welch9). Sequential optimization of computer models based on the expected improvement criteria has proven efficient and effective (see Jones et al.10). These optimization algorithms are global in the sense that they explore regions of the input space in which prediction is poor (potential for optima), while focusing in on regions of space containing optima with high probability. Goodness-of-fit diagnostics, sensitivity analysis and sequential optimization are explored with two analyses of the example using the F-quantile function presented in this article. The first analysis (referred to as