Comment on “Dynamically dimensioned search algorithm for computationally efficient watershed model calibration” by Bryan A. Tolson and Christine A. Shoemaker

[1] Tolson and Shoemaker [2007] recently introduced an optimization algorithm, entitled dynamically dimensioned search (DDS), for automatic calibration of watershedmodels. The DDS method is a simple stochastic neighborhood search algorithm that has been developed with the purpose of finding preferred parameter combinations fast within the user specified maximum number of function evaluations (as opposed to globally optimal solutions). The definition of ‘‘good’’ solution appears somewhat subjective, and has not formally been defined by Tolson and Shoemaker [2007], but refers to the best attainable parameter combination (the lowest value of objective function) for a given number of function evaluations. To benchmark the effectiveness and robustness of DDS, Tolson and Shoemaker [2007] provide a comparison analysis against the shuffled complex evolution (SCE-UA) algorithm previously developed by Duan et al. [1992] for four different optimization problems with increasing complexity. On the basis of this comparison analysis, Tolson and Shoemaker [2007, paragraph 65] conclude that ‘‘the DDS algorithm is a more computationally efficient and robust optimization algorithm than SCEUA in the context of distributed watershed model automatic calibration.’’ We would like to congratulate Tolson and Shoemaker on their paper, which we believe makes a valuable contribution to the field of optimization theory and hydrologic model calibration. However, we wish to communicate some concerns regarding the evaluation methods used in comparison of the SCE-UA and the DDS algorithm. [2] The development of SCE-UA was motivated by the concerns at the time that the commonly available optimization methods (both gradient and nongradient search techniques) were not adequate to address the highly nonlinear, nonconvex, and noncontinuous D-dimensional parameter spaces of typical lumped parameter conceptual watershed models during their calibration phase. This development was warranted and needed, to reduce ambiguity about the optimized parameter estimates (facilitating sound inferences about the system under study), and to obtain a better understanding of the limits of predictive capability of watershed models. Indeed, many contributions to the hydrologic literature and beyond (appropriately acknowledged by Tolson and Shoemaker [2007]), have demonstrated the power and efficiency of the SCE-UA method for finding globally optimal solutions in models with simulation times often on the order of a few seconds or less. The DDS algorithm of Tolson and Shoemaker [2007], however has been developed in the context of finding preferred parameter solutions in computationally demanding models. This context is different than what inspired the original development of the SCE-UA algorithm. Therefore, we believe that any comparison between these methods require careful interpretation and need to be framed within the context of the original goals of the developed algorithms. [3] The difference between the DDS and SCE-UA methods is best highlighted and illustrated in Figure 2 of Tolson and Shoemaker [2007]. For a small budget of function evaluations, it is obvious that the DDS algorithm finds better solutions than the SCE-UA algorithm. However, for larger number of function evaluations, the SCE-UA method generally outperforms the DDS algorithm, locating better overall solutions in the parameter space. One must therefore consider the intended goal, which in the case of DDS is a maximum decline in objective function within a limited but fixed number of function (model) evaluations. For this purpose, we initiated a study related to the efficiency and convergence issues of the SCE-UA algorithm. [4] If the goal is to find preferred solutions fast within a limited budget of model evaluations, the algorithmic parameters of the SCE-UA method might need to be modified in such a way that the method emphasizes less on exhaustive exploration of the global parameter domain, and focuses more on local exploitation of existing solutions. Running SCE-UA with default values of the algorithmic parameters as recommended by Duan et al. [1994], and developed within the context of precisely locating the global optimum, might therefore not be optimal. To improve the initial efficiency of the SCE-UA method, we experimented with Center for Hydrometeorology and Remote Sensing, Henry Samueli School of Engineering, University of California, Irvine, California, USA. Center for Nonlinear Studies, Los Alamos National Laboratory, Los Alamos, New Mexico, USA. Atmospheric, Earth and Energy Department, Lawrence Livermore National Laboratory, Livermore, California, USA.