Why We Really Need to Do Fewer Statistical Tests

I recently wrote an editorial arguing that researchers should run fewer statistical tests (Brenner, 2016). May and Vincent (2017) responded by arguing that we need to switch to using Bayesian tests rather than to doing fewer tests. They claimed that Bayesian methods circumvent the problems that I described because they can be applied to any data at any time. They maintained that it is wasteful to ignore potentially exciting discoveries by not testing them. I will try to explain why using Bayesian statistics does not alleviate the need to run fewer tests. The Bayesian approach has many advantages. The most fundamental advantage is that it allows one to directly judge the likelihood of an effect being present given the data, rather than requiring one to deduce this likelihood from an estimate of the probability of having found such data by chance. One advantage of not having to estimate the latter probability is that one can run the test on the available data without knowing how the experimenter determined when to stop gathering more data (Berger & Berry, 1988). May and Vincent seem to be suggesting that this makes it acceptable to run many tests. However, although the statistical test only depends on the actual data when using Bayesian tests, defining conditions under which data collection will stop influences the data. Thus, relying on Bayesian tests does not mean that you no longer need to consider the way in which the data were acquired. More importantly, my proposal was to reduce the number of hypotheses that are tested, not the number of tests run on a single hypothesis. Irrespective of whether one infers the presence of a certain effect from the fact that a Bayesian test provides strong evidence to support it (Kruschke, 2013; Wagenmakers, 2007) or because a t-test was significant, there is still a chance that there is actually no effect. It is this chance that makes it important to run fewer tests. To understand why, it might help to consider the problem from the point of view of someone reading an article rather than from the point of view of an author. Consider reading an article in which the authors conducted a statistical test and inferred from the outcome of the test that there is an effect. How confident should you be that there really is an effect? To get an idea, consider two overall (unknown) prior probabilities: the probability that there actually is an effect, P(actual), and the probability that the outcome of a statistical test will allow the authors to infer that there is an effect, P(inferred). These two probabilities are hopefully not independent, because we hope that the outcome of a statistical test tells us something about the probability of the effect being present, so we relate them with Bayes’ rule.