Challenges and suggestions for defining replication "success" when effects may be heterogeneous: Comment on Hedges and Schauer (2019).
Challenges and suggestions for defining replication "success" when effects may be heterogeneous: Comment on Hedges and Schauer (2019).
复制标题
当效果可能不同时定义复制“成功”的挑战和建议：对 Hedges 和 Schauer 的评论（2019）。
DOI：
10.1037/met0000223
复制
发表时间：
2019
影响因子：
7
通讯作者：
VanderWeele,TylerJ
中科院分区：
文献类型：
--
作者：
Mathur,MayaB;VanderWeele,TylerJ
心理学家现在正试图从头开始复制已发表的研究，以证实这些发现。在一个日益广泛的复制研究设计中，几个合作站点（如大学）中的每一个都试图独立地复制一个原始研究，并且结果是跨站点综合的。Hedges和Schauer（2019）对这些复制项目提出了统计分析；他们的分析侧重于通过在一组重复研究中测试异质性，同时排除原始研究，来评估结果在不同复制地点之间的差异程度。我们同意他们关于现有分析方法的局限性和考虑重复异质性的重要性的前提。这个目标本身可能很有趣。然而，我们认为，通过只关注重复性研究是否具有彼此相似的效应大小，这些分析并不特别适合评估重复是否实际上支持所调查的科学效果或评估多地点复制项目的能力。我们使用直接解决这些目标的复制成功的替代指标重新分析了Hedges和Schauer（2019）的示例数据集。对于复制成功，我们得出了比他们更乐观的结论，这表明替代指标可能导致与Hedges和Schauer（2019）截然不同的结论。
Psychological scientists are now trying to replicate published research from scratch to confirm the findings. In an increasingly widespread replication study design, each of several collaborating sites (such as universities) independently tries to replicate an original study, and the results are synthesized across sites. Hedges and Schauer (2019) proposed statistical analyses for these replication projects; their analyses focus on assessing the extent to which results differ across the replication sites, by testing for heterogeneity among a set of replication studies, while excluding the original study. We agree with their premises regarding the limitations of existing analysis methods and regarding the importance of accounting for heterogeneity among the replications. This objective may be interesting in its own right. However, we argue that by focusing only on whether the replication studies have similar effect sizes to one another, these analyses are not particularly appropriate for assessing whether the replications in fact support the scientific effect under investigation or for assessing the power of multisite replication projects. We reanalyze Hedges and Schauer’s (2019) example dataset using alternative metrics of replication success that directly address these objectives. We reach a more optimistic conclusion regarding replication success than they did, illustrating that the alternative metrics can lead to quite different conclusions from those of Hedges and Schauer (2019).