Open AccessContext Dependency as a Predictor of Replicability
Author(s) -
Mario Gollwitzer,
Johannes Schwabe
Publication year - 2021
Publication title -
review of general psychology
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.519
H-Index - 98
eISSN - 1939-1552
pISSN - 1089-2680
DOI - 10.1177/10892680211015635
Subject(s) - context (archaeology) , replication (statistics) , argument (complex analysis) , psychology , dependency (uml) , affect (linguistics) , cognitive psychology , context effect , social psychology , epistemology , computer science , statistics , artificial intelligence , mathematics , philosophy , communication , word (group theory) , biology , paleontology , biochemistry , chemistry , geometry
We scrutinize the argument that unsuccessful replications—and heterogeneous effect sizes more generally—may reflect an underappreciated influence of context characteristics. Notably, while some of these context characteristics may be conceptually irrelevant (as they merely affect psychometric properties of the measured/manipulated variables), others are conceptually relevant as they qualify a theory. Here, we present a conceptual and analytical framework that allows researchers to empirically estimate the extent to which effect size heterogeneity is due to conceptually relevant versus irrelevant context characteristics. According to this framework, contextual characteristics are conceptually relevant when the observed heterogeneity of effect sizes cannot be attributed to psychometric properties. As an illustrative example, we demonstrate that the observed heterogeneity of the “moral typecasting” effect, which had been included in the ManyLabs 2 replication project, is more likely attributable to conceptually relevant rather than irrelevant context characteristics, which suggests that the psychological theory behind this effect may need to be specified. In general, we argue that context dependency should be taken more seriously and treated more carefully by replication research.