Multiple Inferences Using Confidence Intervals

SUMMARY 1. In a recent review article, the problem of making false‐ positive inferences as a result of making multiple comparisons between groups of experimental units or between experimental outcomes was addressed. 2. It was concluded that the most universally applicable solution was to use the Ryan–Holm step‐down Bonferroni procedure to control the family‐wise (experiment‐wise) type 1 error rate. This procedure consists of adjusting the P values resulting from hypothesis testing. It allows for correlation among hypotheses and has been validated by Monte Carlo simulation. It is a simple procedure and can be performed by hand. 3. However, some investigators prefer to estimate effect sizes and make inferences by way of confidence intervals rather than, or in addition to, testing hypotheses by way of P values and it is the policy of some editors of biomedical journals to insist on this. It is not generally recognized that confidence intervals, like P values, must be adjusted if multiple inferences are made from confidence intervals in a single experiment. 4. In the present review, it is shown how confidence intervals can be adjusted for multiplicity by an extension of the Ryan–Holm step‐down Bonferroni procedure. This can be done for differences between group means in the case of continuous variables and for odds ratios or relative risks in the case of categorical variables set out as 2 × 2 tables.