Premium
Discrimination slope and integrated discrimination improvement – properties, relationships and impact of calibration
Author(s) -
Pencina Michael J.,
Fine Jason P.,
D'Agostino Ralph B.
Publication year - 2016
Publication title -
statistics in medicine
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.996
H-Index - 183
eISSN - 1097-0258
pISSN - 0277-6715
DOI - 10.1002/sim.7139
Subject(s) - brier score , metric (unit) , regret , mathematics , measure (data warehouse) , calibration , statistics , sensitivity (control systems) , function (biology) , mean squared error , event (particle physics) , regression , econometrics , computer science , data mining , operations management , physics , quantum mechanics , electronic engineering , evolutionary biology , biology , engineering , economics
Discrimination slope, defined as the slope of a linear regression of predicted probabilities of event derived from a prognostic model on the binary event status, has recently gained popularity as a measure of model performance. It is as a building block for the integrated discrimination improvement that equals the difference in discrimination slopes between the two models being compared. Several authors have pointed out that it does not make sense to apply the integrated discrimination improvement and discrimination slope when working with mis‐calibrated models, whereas others have raised concerns about the ability of improving discrimination slope without adding new information. In this paper, we show that under certain assumptions the discrimination slope is asymptotically related to two other R‐squared measures, one of which is a rescaled version of the Brier score, known to be proper. Furthermore, we illustrate how a simple recalibration makes the slope equal to the rescaled Brier R‐squared metric. We also show that the discrimination slope can be interpreted as a measure of reduction in expected regret for the Gini‐Brier regret function. Using theoretical and practical examples, we illustrate how all of these metrics are affected by different levels of model mis‐calibration. In particular, we demonstrate that simple recalibration ascertaining calibration in‐the‐large and calibration slope equal to 1 are not sufficient to correct for some forms of mis‐calibration. We conclude that R‐squared metrics, including the discrimination slope, offer an attractive choice for quantifying model performance as long as one accounts for their sensitivity to model calibration. Copyright © 2016 John Wiley & Sons, Ltd.