Resampling‐based stepwise multiple testing procedures with applications to clinical trial data

SUMMARY Endpoints in clinical trials are often highly correlated. However, the commonly used multiple testing procedures in clinical trials either do not take into consideration the correlations among test statistics or can only exploit known correlations. Westfall and Young constructed a resampling‐based stepdown method that implicitly utilizes the correlation structure of test statistics in situations with unknown correlations. However, their method requires a “subset pivotality” assumption. Romano and Wolf proposed a more general stepdown method, which does not require such an assumption. There is at present little experience with the application of such methods in analyzing clinical trial data. We advocate the application of resampling‐based multiple testing procedures to clinical trials data when appropriate. We have conjectured that the resampling‐based stepdown methods can be extended to a stepup procedure under appropriate assumptions and examined the performance of both stepdown and stepup methods under a variety of correlation structures and distribution types. Results from our simulation studies support the use of the resampling‐based methods under various scenarios, including binary data and small samples, with strong control of Family wise type I error rate (FWER). Under positive dependence and for binary data even under independence, the resampling‐based methods are more powerful than the Holm and Hochberg methods. Last, we illustrate the advantage of the resampling‐based stepwise methods with two clinical trial data examples: a cardiovascular outcome trial and an oncology trial.