Valid randomization‐based p ‐values for partially post hoc subgroup analyses

By ‘partially post‐hoc’ subgroup analyses, we mean analyses that compare existing data from a randomized experiment—from which a subgroup specification is derived—to new, subgroup‐only experimental data. We describe a motivating example in which partially post hoc subgroup analyses instigated statistical debate about a medical device's efficacy. We clarify the source of such analyses' invalidity and then propose a randomization‐based approach for generating valid posterior predictive p ‐values for such partially post hoc subgroups. Lastly, we investigate the approach's operating characteristics in a simple illustrative setting through a series of simulations, showing that it can have desirable properties under both null and alternative hypotheses. Copyright © 2015 John Wiley & Sons, Ltd.