Reply to Gelman, Gaudart, Pericchi: More reasons to revise standards for statistical evidence

I thank the editor for selecting these letters for discussion; together, they typify the most common objections to raising standards for statistical evidence. Gelman and Robert’s (1) letter characterizes subjective Bayesian objections to the use of more stringent statistical standards, arguing that significance levels and evidence thresholds should be based on “costs, benefits, and probabilities of all outcomes.” In principle, this is a wonderful goal, but in practice, it is impossible to achieve. In most hypothesis tests, unique and well-defined loss functions and prior densities do not exist. Instead, a plethora of vaguely defined loss functions and prior densities exist. In the context of declaring a scientific discovery, distinct loss functions and prior beliefs are held by the study investigators, by journal editors and reviewers, by other scientists, by funding agencies, and by the general public. Thousands of scientific manuscripts are written each year, and eliciting these distinct loss functions and priors on a case-by-case basis, and determining how to combine them, is simply not feasible. Worse still, the authors of each manuscript select from multiple hypothesis tests to report. This is why hypothesis tests are usually performed using commonly accepted standards for statistical significance. With regard to loss functions, it is important to note that the greatest loss that the scientific community now faces is the loss of public confidence. Under a prior assumption of equipoise, the declaration of new discoveries based on P values near 0.05 guarantees that ∼20% of these discoveries will be false. Even higher rates of nonreproducibility were found in the empirical studies cited in ref. 2, suggesting that the assumption of equipoise is also overly optimistic. Such rates of nonreproducibility are clearly too high to maintain public confidence in science. Gaudart et al. (3) describe a second general objection to raising the bar …