Statistics, Handle with Care: Detecting Multiple Model Components with the Likelihood Ratio Test

The likelihood ratio test (LRT) and the related $F$ test, do not (evenasymptotically) adhere to their nominal $\chi^2$ and $F$ distributions in manystatistical tests common in astrophysics, thereby casting many marginal line orsource detections and non-detections into doubt. Although there are manylegitimate uses of these statistics, in some important cases it can beimpossible to compute the correct false positive rate. For example, it hasbecome common practice to use the LRT or the $F$ test for detecting a line in aspectral model or a source above background despite the lack of certainrequired regularity conditions. In these and other settings that involvetesting a hypothesis that is on the boundary of the parameter space, {\itcontrary to common practice, the nominal $\chi^2$ distribution for the LRT orthe $F$ distribution for the $F$ test should not be used}. In this paper, wecharacterize an important class of problems where the LRT and the $F$ test failand illustrate this non-standard behavior. We briefly sketch several possibleacceptable alternatives, focusing on Bayesian posterior predictiveprobability-values. We present this method in some detail, as it is a simple,robust, and intuitive approach. This alternative method is illustrated usingthe gamma-ray burst of May 8, 1997 (GRB 970508) to investigate the presence ofan Fe K emission line during the initial phase of the observation.Comment: Twenty four pages, seven figures. The Astrophysical Journal, May 2002, to appea