Randomization inference for treatment effects on a binary outcome

Two methods are developed for constructing randomization‐based confidence sets for the average effect of a treatment on a binary outcome. The methods are nonparametric and require no assumptions about random sampling from a larger population. Both of the resulting 1 − α confidence sets are exact in the sense that the probability of containing the true treatment effect is at least 1 − α . Both types of confidence sets are also guaranteed to have width no greater than one. In contrast, a previously proposed asymptotic confidence interval is not exact and may have width greater than 1. The first approach combines Bonferroni‐adjusted prediction sets for the attributable effects in the treated and untreated. The second method entails inverting a permutation test. Simulations are presented comparing the two randomization‐based confidence sets with the asymptotic interval as well as the standard Wald confidence interval and a commonly used exact interval for the difference in binomial proportions. Results show for small to moderate sample sizes that the permutation confidence set attains the narrowest width on average among the methods that maintain nominal coverage. Extensions that allow for stratifying on categorical baseline covariates are also discussed. Copyright © 2014 John Wiley & Sons, Ltd.