Improving the replicability of neuroimaging findings by thresholding effect sizes instead of p ‐values

Abstract The classical approach for testing statistical images using spatial extent inference (SEI) thresholds the statistical image based on the p ‐value. This approach has an unfortunate consequence on the replicability of neuroimaging findings because the targeted brain regions are affected by the sample size—larger studies have more power to detect smaller effects. Here, we use simulations based on the preprocessed Autism Brain Imaging Data Exchange (ABIDE) to show that thresholding statistical images by effect sizes has more consistent estimates of activated regions across studies than thresholding by p ‐values. Using a constant effect size threshold means that the p ‐value threshold naturally scales with the sample size to ensure that the target set is similar across repetitions of the study that use different sample sizes. As a consequence of thresholding by the effect size, the type 1 and type 2 error rates go to zero as the sample size gets larger. We use a newly proposed robust effect size index that is defined for an arbitrary statistical image so that effect size thresholding can be used regardless of the test statistic or model.