A Unified Framework to Quantify the Credibility of Scientific Findings
A Unified Framework to Quantify the Credibility of Scientific Findings
复制标题
DOI：
10.1177/2515245918787489
复制
发表时间：
2018-09-01
影响因子：
13.6
通讯作者：
Vanpaemel, Wolf
中科院分区：
文献类型：
--
作者：
LeBel, Etienne P.;McCarthy, Randy J.;Vanpaemel, Wolf
社会投资于科学研究，以更好地了解世界，并试图利用这种改进的理解来解决紧迫的社会问题。然而，发表的研究只有在可信的情况下，才能对理论或应用有用。在科学领域，一个可信的发现是一个反复经受住危险的伪造企图的发现。然而，最先进的荟萃分析方法无法确定效果的可信度，因为它们没有考虑到每个纳入的研究在多大程度上幸免于这种伪造行为。为了克服这个问题，我们概述了一个统一的框架，通过检查与可证伪性相关的四个基本维度来估计已发表研究的可信度：(a)方法和数据的透明度，(b)重新应用相同的数据处理和分析决策时结果的可重复性，(c)结果对不同数据处理和分析决策的稳健性，以及(d)效果的可复制性。这个框架包括一个标准化的工作流程，在这个工作流程中，一个发现通过仔细审查的程度是按照可信度的这四个方面进行量化的。该框架通过将其应用于心理学文献中已发表的重复研究来证明。最后，我们概述了该框架的Web实现，并鼓励研究人员社区为该平台的开发和众包做出贡献。
Societies invest in scientific studies to better understand the world and attempt to harness such improved understanding to address pressing societal problems. Published research, however, can be useful for theory or application only if it is credible. In science, a credible finding is one that has repeatedly survived risky falsification attempts. However, state-of-the-art meta-analytic approaches cannot determine the credibility of an effect because they do not account for the extent to which each included study has survived such attempted falsification. To overcome this problem, we outline a unified framework for estimating the credibility of published research by examining four fundamental falsifiability-related dimensions: (a) transparency of the methods and data, (b) reproducibility of the results when the same data-processing and analytic decisions are reapplied, (c) robustness of the results to different data-processing and analytic decisions, and (d) replicability of the effect. This framework includes a standardized workflow in which the degree to which a finding has survived scrutiny is quantified along these four facets of credibility. The framework is demonstrated by applying it to published replications in the psychology literature. Finally, we outline a Web implementation of the framework and conclude by encouraging the community of researchers to contribute to the development and crowdsourcing of this platform.