Premium
A note on the evaluation of novel biomarkers: do not rely on integrated discrimination improvement and net reclassification index
Author(s) -
Hilden Jørgen,
Gerds Thomas A.
Publication year - 2013
Publication title -
statistics in medicine
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.996
H-Index - 183
eISSN - 1097-0258
pISSN - 0277-6715
DOI - 10.1002/sim.5804
Subject(s) - brier score , computer science , index (typography) , statistics , data mining , measure (data warehouse) , outcome (game theory) , machine learning , econometrics , artificial intelligence , mathematics , mathematical economics , world wide web
The ‘integrated discrimination improvement’ (IDI) and the ‘net reclassification index’ (NRI) are statistics proposed as measures of the incremental prognostic impact that a new biomarker will have when added to an existing prediction model for a binary outcome. By design, both measures were meant to be intuitively appropriate, and the IDI and NRI formulae do look intuitively plausible. Both have become increasingly popular. We shall argue, however, that their use is not always safe. If IDI and NRI are used to measure gain in prediction performance, then poorly calibrated models may appear advantageous, and in a simulation study, even the model that actually generates the data (and hence is the best possible model) can be improved on without adding measured information. We illustrate these shortcomings in actual cancer data as well as by Monte Carlo simulations. In these examples, we contrast IDI and NRI with the area under ROC and the Brier score. Unlike IDI and NRI, these traditional measures have the characteristic that prognostic performance cannot be accidentally or deliberately inflated. Copyright © 2013 John Wiley & Sons, Ltd.