The design of replication studies
The design of replication studies
复制标题
DOI：
10.1111/rssa.12688
复制
发表时间：
2021-03-31
影响因子：
2
通讯作者：
Schauer, Jacob M.
中科院分区：
文献类型：
--
作者：
Hedges, Larry V.;Schauer, Jacob M.
对复制的经验评估已经变得越来越普遍，但还没有统一的方法来这样做。一些评估只进行一次重复研究，而另一些评估则进行多项研究，通常跨越多个实验室。设计这样的程序在很大程度上需要解决一些棘手的问题，即一组研究需要哪些实验部分才能被认为是重复的。然而，另一个重要的考虑因素是，重复研究的设计应支持足够敏感的分析。例如，如果要对复制进行假设测试，则应设计研究以确保这些测试是有效的；如果不是，可能很难最终确定复制尝试是成功还是失败。本文描述了设计复制研究集合的方法，以确保它们既足够敏感又具有成本效益。它描述了重复研究的两种潜在分析--假设检验和方差分量估计--以及为它们获得最优设计的方法。利用这些结果，它评估了许多实验室项目使用的统计能力、点估计器的精确度和设计的最佳性，并发现虽然它可能已经足够强大来检测研究之间的一些较大差异，但其他设计将成本更低和/或产生更精确的估计或更强大的假设检验。
Empirical evaluations of replication have become increasingly common, but there has been no unified approach to doing so. Some evaluations conduct only a single replication study while others run several, usually across multiple laboratories. Designing such programs has largely contended with difficult issues about which experimental components are necessary for a set of studies to be considered replications. However, another important consideration is that replication studies be designed to support sufficiently sensitive analyses. For instance, if hypothesis tests are to be conducted about replication, studies should be designed to ensure these tests are well-powered; if not, it can be difficult to determine conclusively if replication attempts succeeded or failed. This paper describes methods for designing ensembles of replication studies to ensure that they are both adequately sensitive and cost-efficient. It describes two potential analyses of replication studies-hypothesis tests and variance component estimation-and approaches to obtaining optimal designs for them. Using these results, it assesses the statistical power, precision of point estimators and optimality of the design used by the Many Labs Project and finds that while it may have been sufficiently powered to detect some larger differences between studies, other designs would have been less costly and/or produced more precise estimates or higher-powered hypothesis tests.