Open AccessUnderstanding forecast verification statistics
Author(s) -
Mason S. J.
Publication year - 2008
Publication title -
meteorological applications
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.672
H-Index - 59
eISSN - 1469-8080
pISSN - 1350-4827
DOI - 10.1002/met.51
Subject(s) - computer science , confidence interval , strengths and weaknesses , probabilistic logic , quality (philosophy) , selection (genetic algorithm) , plea , statistics , sample size determination , sample (material) , scoring rule , bayesian probability , econometrics , artificial intelligence , machine learning , mathematics , psychology , philosophy , chemistry , epistemology , law , social psychology , chromatography , political science
Although there are numerous reasons for performing a verification analysis, there are usually two general questions that are of interest: are the forecasts good, and can we be confident that the estimate of forecast quality is not misleading? When calculating a verification score, it is not usually obvious how the score can answer either of these questions. Some procedures for attempting to answer the questions are reviewed, with particular focus on p ‐values and confidence intervals. P ‐values are shown to be rather unhelpful in answering either question, especially when applied to probabilistic verification scores, and confidence intervals are to be preferred. However, confidence intervals cannot reveal biases in the value of a score that arises from an inadequate experimental design for testing on truly out‐of‐sample observations. Some specific problems with cross validation are highlighted. Finally, in the interests of increasing the insight into forecast strengths and weaknesses and in pointing towards methods for improving forecast quality, a plea is made for a more discriminating selection of verification procedures than has been adopted to date. Copyright © 2008 Royal Meteorological Society