Statistical Principles: Myths or Facts?

geneity, pooled confidence interval well away from zero, a justification that a biased selection of studies and/or endpoints is unlikely, and a sensitivity analysis demonstrating the robustness of the findings. They specifically state that a retrospective meta-analysis of only two studies originally intended to stand on their own will not generally be useful and that a metaanalysis cannot be used to reconcile the results of two conflicting studies, one which is positive and one which is negative. The clinical relevance of the findings, and the heterogeneity and external validity of the results must also be taken into account. Properly done meta-analyses based on individual patient data from all randomized clinical trials dealing with a specific treatment comparison provide, in fact, the highest level of evidence-based medicine. They yield the best overall estimate of the size of the treatment effect, a confidence interval for the size of the difference, along with an assessment of the heterogeneity of results between the studies and eventually across the levels of different prognostic factors. Two examples are the adjuvant tamoxifen meta-analysis in early breast cancer [3] and the more recent meta-analysis of the effect of bacillus Calmette-Guerin on progression in superficial bladder cancer [4]. Few of the individual trials in either meta-analysis showed a significant treatment benefit, but the overall results from each meta-analysis provided quite striking differences in treatment efficacy which have had an important impact on clinical practice. Thus it is important to properly assess all the available evidence at hand and not to concentrate on whether one or two trials may or may not show statistically significant differences. The other myth that I would like to comment on is that of ‘one hypothesis, one trial design’. The example given is a 3-arm trial where arm A is a new treatment, arm B is the standard treatment in standard duration, and arm C is the standard treatment in shorter duration. This example raises the question of the appropriate significance levels to use when faced with multiple testing. In an interesting paper in this issue of ONKOLOGIE, Edler and Kopp-Schneider [1] discuss a number of ‘myths’ concerning the principles involved in the statistical design and interpretation of the results of randomized phase III clinical trials. I would like to briefly comment on two of these myths. The first myth which is rejected by the authors is that two independent positive studies are necessary to claim treatment efficacy. The authors suggest there may be instances where one study might be sufficient or that one could combine the results of two different studies. The Committee for Proprietary Medicinal Products (CPMP) has provided guidelines on this subject: Points to Consider on Validity and Interpretation of Meta-Analyses, and One Pivotal Study [2]. Although in most drug applications the proof of efficacy relies on several studies, the CPMP recognizes that there are cases where applications may be based on the results from one study or on the pooled results from several studies (meta-analyses). Although it is usually wise to plan more than one study in the phase III program of a new drug, there is no formal requirement to carry out two or more pivotal studies. The minimum requirement set forth by the CPMP [2] is one controlled study with statistically compelling and clinically relevant results. The following factors should be considered: the trial’s internal and external validity, clinical relevance, degree of statistical relevance, data quality, internal consistency, center effects, and the plausibility of the hypothesis tested. Thus, a single, high-quality, large, multi-center trial might provide sufficient evidence of treatment efficacy if the treatment results are statistically compelling, clinically relevant, and reasonably homogeneous across both institutions and prognostic factors. Although meta-analyses may be problematic for a number of reasons, the CPMP has also spelled out the conditions under which meta-analyses may provide pivotal evidence [2]. While meta-analyses should preferably be prospectively defined, prerequisites for a potentially acceptable retrospective metaanalysis include: some studies clearly positive, inconclusive studies showing positive trends, no statistical or major hetero-