Premium
On assessing calibration of multivariate ensemble forecasts
Author(s) -
Wilks Daniel S.
Publication year - 2016
Publication title -
quarterly journal of the royal meteorological society
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.744
H-Index - 143
eISSN - 1477-870X
pISSN - 0035-9009
DOI - 10.1002/qj.2906
Subject(s) - univariate , multivariate statistics , calibration , curse of dimensionality , covariance , ensemble forecasting , computer science , rank (graph theory) , multivariate analysis , statistics , econometrics , mathematics , artificial intelligence , combinatorics
A necessary condition for optimal use in decision making of probability forecasts in general, and ensemble forecasts in particular, is that they should be calibrated. Ensemble calibration implies that each observation is statistically indistinguishable from the members of its forecast ensemble. The dimensionality of multivariate forecast settings, where the observation is a vector of related quantities rather than a scalar, affords opportunities for a rich variety of types of ensemble miscalibration, and so requires more sophisticated metrics for detection and diagnosis. This article investigates the performance in an artificial data setting of five multivariate ensemble calibration metrics that have been proposed in the literature. It is found that none of these dominates the others for detecting covariance miscalibration across a variety of miscalibration types, and that each exhibits weak performance in some instances, implying that several of the metrics should be used simultaneously when multivariate ensemble miscalibration is to be investigated. Multivariate biases were detected more strongly and diagnostically using univariate rank histograms for the individual forecast vector components.