The Replication Paradox: Combining Studies can Decrease Accuracy of Effect Size Estimates

Replication is often viewed as the demarcation between science and nonscience. However,contrary to the commonly held view, we show that in the current (selective) publicationsystem replications may increase bias in effect size estimates. Specifically, we examinethe effect of replication on bias in estimated population effect size as a function ofpublication bias and the studies’ sample size or power. We analytically show thatincorporating the results of published replication studies will in general not lead toless bias in the estimated population effect size. We therefore conclude that merereplication will not solve the problem of overestimation of effect sizes. We will discussthe implications of our findings for interpreting results of published and unpublishedstudies, and for conducting and interpreting results of meta-analyses. We also discusssolutions for the problem of overestimation of effect sizes, such as discarding and notpublishing small studies with low power, and implementing practices that completelyeliminate publication bias (e.g., study registration).