Open AccessPerformance targets and the Brier score
Author(s) -
Roulston Mark S.
Publication year - 2007
Publication title -
meteorological applications
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.672
H-Index - 59
eISSN - 1469-8080
pISSN - 1350-4827
DOI - 10.1002/met.21
Subject(s) - brier score , scoring rule , computer science , econometrics , forecast skill , function (biology) , hedge , actuarial science , statistics , economics , artificial intelligence , mathematics , machine learning , ecology , evolutionary biology , biology
The Brier score and Brier skill score are commonly used measures for evaluating probability forecasts. An oft‐cited desirable property of these measures is that they are strictly proper, meaning that forecasters wishing to optimise the expected value of these measures should report their true beliefs and not ‘hedge’ the forecast by reporting probabilities that differ from their true beliefs. In this article, it is argued that the propriety of the Brier score only discourages hedging if the utility of the forecaster is a linear function of the score. In particular, it is demonstrated, that if known performance targets based on the score are imposed on the forecaster then this negates the propriety of the score. Forecasters wishing to maximise the probability of achieving the target value of the Brier score or Brier skill score should hedge their forecasts rather than report their true beliefs. In contrast, performance targets drawn from a uniform distribution which are unknown to the forecaster during the evaluation period will induce the utility function of the forecaster to be a linear function of the Brier score. This preserves the propriety of the scoring rule. © Crown Copyright 2007. Reproduced with the permission of the Controller of HMSO. Published by John Wiley & Sons, Ltd