Evaluating technologies for classification and prediction in medicine

Abstract Modern technologies promise to provide new ways of diagnosing disease, detecting subclinical disease, predicting prognosis, selecting patient specific treatment, identifying subjects at risk for disease, and so forth. Advances in genomics, proteomics and imaging modalities in particular hold great potential for assisting with classification/prediction in medicine. Before a classifier can be adopted for routine use in health care, its classification accuracy must be determined. Standards for evaluating new clinical classifiers however, lag far behind the well established standards that exist for evaluating new clinical treatments. In this paper, we discuss a phased approach to developing a new classifier (or biomarker). It mirrors the internationally established phase 1–2–3 paradigm for therapeutic drugs. The defined phases lead to a logical sequence of studies for classifier development. We emphasize that evaluating classification accuracy is fundamentally different from simply establishing association with outcome. Therefore, study objectives and designs differ from the familiar methods of clinical trials. We discuss these briefly for each phase. Finally, we argue that classifier development requires some rethinking of traditional data analysis techniques. As an example we show that maximizing the likelihood function to fit a logistic regression model to multiple predictors, can yield a poor classifier. Instead we demonstrate that an approach that maximizes an alternative objective function characterizing classification accuracy performs better. Copyright © 2005 John Wiley & Sons, Ltd.