Premium
ASSESSING ERROR RATE ESTIMATORS: THE LEAVE‐ONE‐OUT METHOD RECONSIDERED
Author(s) -
Krzanowski W.J.,
Hand D.J.
Publication year - 1997
Publication title -
australian journal of statistics
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.434
H-Index - 41
eISSN - 1467-842X
pISSN - 0004-9581
DOI - 10.1111/j.1467-842x.1997.tb00521.x
Subject(s) - estimator , statistics , bias of an estimator , bayes' theorem , efficient estimator , minimum variance unbiased estimator , econometrics , word error rate , stein's unbiased risk estimate , computer science , mathematics , bayesian probability , artificial intelligence
summary Many comparative studies of the estimators of error rates of supervised classification rules are based on inappropriate criteria. In particular, although they fix the Bayes error rate, their summary statistics aggregate a range of true error rates. This means that their conclusions about the performance of classification rules cannot be trusted. This paper discusses the general issues involved, and then focuses attention specifically on the leave‐one‐out estimator. The estimator is investigated in a simulation study, both in absolute terms and in comparison with a popular bootstrap estimator. An improvement to the leave‐one‐out estimator is suggested, but the bootstrap estimator appears to maintain superiority even when the criteria are adjusted.