Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature.
Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature.
复制标题
DOI：
10.1371/journal.pbio.2000797
复制
发表时间：
2017-03
期刊：
影响因子：
9.8
通讯作者：
Ioannidis JP
中科院分区：
文献类型：
--
作者：
Szucs D;Ioannidis JP
我们通过分析最近发表的3，801篇认知神经科学和心理学论文中的26，841条统计记录，对已发表的效应量分布和估计功效进行了实证评估。对于名义上具有统计学显著性的结果，报告的中位效应量为D = 0.93（四分位距：0.64-1.46），对于非显著性结果，报告的中位效应量为D = 0.24（0.11-0.42）。检测小、中、大效应的中位把握度分别为0.12、0.44和0.73，反映了过去半个世纪没有改善。这是因为样本量仍然很小。假设两个学科的真实效应大小相似，认知神经科学的功效低于心理学。期刊影响因子与影响力呈负相关。假设零假设的先验概率在一个现实的范围内，整个文献的误报概率可能超过50%。根据我们的研究结果，最近报道的心理学低复制成功率是现实的，认知神经科学的表现可能会更差。生物医学、心理学和许多其他领域可能正遭受严重的复制危机。为了深入了解这场危机背后的一些因素，我们分析了从数千篇认知神经科学和心理学研究论文中提取的统计信息。我们确定，在过去的半个世纪，发现现有关系的统计能力并没有提高。统计功效低的一个后果是，研究可能会报告许多假阳性结果。使用我们的大型数据集，我们估计了统计学上显著的发现是错误的概率（称为错误报告概率）。通过对研究人员提出正确假设的频率进行一些合理假设，我们得出结论，超过50%的被认为具有统计学意义的已发表研究结果可能是错误的。我们还观察到，由于认知神经科学的样本量较小，认知神经科学研究的错误报告概率高于心理学研究。此外，发表研究的期刊的影响因子越高，统计功效就越低。根据我们的研究结果，最近报道的心理学低复制成功率是现实的，认知神经科学的表现可能会更差。
We have empirically assessed the distribution of published effect sizes and estimated power by analyzing 26,841 statistical records from 3,801 cognitive neuroscience and psychology papers published recently. The reported median effect size was D = 0.93 (interquartile range: 0.64–1.46) for nominally statistically significant results and D = 0.24 (0.11–0.42) for nonsignificant results. Median power to detect small, medium, and large effects was 0.12, 0.44, and 0.73, reflecting no improvement through the past half-century. This is so because sample sizes have remained small. Assuming similar true effect sizes in both disciplines, power was lower in cognitive neuroscience than in psychology. Journal impact factors negatively correlated with power. Assuming a realistic range of prior probabilities for null hypotheses, false report probability is likely to exceed 50% for the whole literature. In light of our findings, the recently reported low replication success in psychology is realistic, and worse performance may be expected for cognitive neuroscience. Biomedical science, psychology, and many other fields may be suffering from a serious replication crisis. In order to gain insight into some factors behind this crisis, we have analyzed statistical information extracted from thousands of cognitive neuroscience and psychology research papers. We established that the statistical power to discover existing relationships has not improved during the past half century. A consequence of low statistical power is that research studies are likely to report many false positive findings. Using our large dataset, we estimated the probability that a statistically significant finding is false (called false report probability). With some reasonable assumptions about how often researchers come up with correct hypotheses, we conclude that more than 50% of published findings deemed to be statistically significant are likely to be false. We also observed that cognitive neuroscience studies had higher false report probability than psychology studies, due to smaller sample sizes in cognitive neuroscience. In addition, the higher the impact factors of the journals in which the studies were published, the lower was the statistical power. In light of our findings, the recently reported low replication success in psychology is realistic, and worse performance may be expected for cognitive neuroscience.