Acknowledge and fix the multiple testing problem

Vandenbroucke 1 comments 'Observational research that is started with equally high priors can be expected to perform as well as randomized trials.' Empirical evidence is on the other side. It is not just hormone replacement therapy that has come a cropper. What about vitamins E and C, and selenium and multivitamins, to name but a few? 2–4 Maziak 5 points to what appears to be a general problem: claims coming from observational studies generally fail to replicate. See also Ioannidis 6 and Boffetta et al. 7 One small step forward would be to acknowledge that multiple testing is likely part of the problem. How can you ask tens to hundreds of questions and not expect false positives? Those that argue for no correction for multiple testing 8,9 leave me not persuaded. One purpose of a scientific paradigm is to lead to good practice and empirically verified results. Why continue a data analysis strategy that is wrong 80–90% of the time? Multiple testing should be fixed before someone says, why fund any of this research? References 1 Vandenbroucke JP. Commentary: Maziak's essay, seen from another angle. Lipworth L, Blot WJ. False-positive results in cancer epidemiology: a plea for epistemological modesty. Dr Young has one cogent argument: he is 'not per-suaded' by those that argue for no correction for multiple testing. Fine, but what are his reasons? What is, for example, the difference between genome-wide testing of the association of a few hundred thousand SNPs with disease, and a PhD student who uses data collected for another purpose to see whether new findings by other researchers hold? The latter student is also part of the game of asking 'hundreds of ques-tions'. The main examples that Young cites, Vitamin C, E, selenium etc. have almost certainly nothing to do with the multiple testing problem. The statement that observational studies 'generally fail to replicate' is unsubstantiated. Moreover, failure of replication is expected when explanatory science develops, and should be more frequent for etiologic research than for randomized trials, as I have argued in the paper that left Dr Young not persuaded. 1 Finally, generalizations about observational research are unwarranted: all genetics is observational research, so are all infectious disease outbreak investigations, all study of prognosis is observational, as are most studies of diagnosis. It is more meaningful to think separately about different fields of observational research—e.g. for adverse effects research, there is no systematic …