Improving Reproducibility by Incorporating Uncertainty

通过纳入不确定性来提高再现性

• 批准号: 10322751
• 负责人: Timothy L. Lash
• 金额: $ 27.05万
• 依托单位: EMORY UNIVERSITY
• 依托单位国家: 美国
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-02-06 至 2024-01-31
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/10322751
• 关键词: Address; Appearance; Area; Bayesian Method; Behavior; Body mass index; CYP2D6 gene; Clinical; Communities; Consensus; Creativeness; Data Collection; Effectiveness; Evaluation; Genotype; Grant; Guidelines; Heterogeneity; Individual; Influentials; Informatics; Journals; Judgment; Life Style; Measurement; Meta-Analysis; Methods; Obesity; Outcome; Paper; Perception; Predisposition; Publication Bias; Reaction; Relative Risks; Reporting; Reproducibility; Research; Research Personnel; Residual state; Resources; Risk; Selection Bias; Source; Speed; Systematic Bias; Tamoxifen; Techniques; Testing; Uncertainty; United States National Institutes of Health; advanced analytics; base; cancer recurrence; complex data; disorder later incidence prevention; editorial; evidence base; improved; informatics tool; malignant breast neoplasm; metabolic phenotype; mortality; simulation; skills; web-enabled

In recent years, stakeholders in the scientific and lay communities have raised alarms about a lack of reproducibility of scientific results. These stakeholders view the reproducibility crisis as a product of the behavior of researchers and editors. While these behaviors likely have an impact on reproducibility, there are credible reasons why studies of complex data should be expected to arrive at different estimates. First, each study is differentially susceptible to systematic biases, including confounding, selection bias and measurement error. These biases may be large drivers of the appearance of poor reproducibility. Second, many studies that have been criticized for lack of replication are small, and therefore subject to substantial random variability. In combination with selection forces emanating from significance testing, these small studies are likely to overestimate effects, further contributing to the appearance of poor reproducibility. To date, proposed solutions to the perceived reproducibility crisis have largely ignored these contributing factors. We propose to (a) use simulation-based quantitative bias analysis techniques to adjust for the influence of systematic errors on estimates of association and on summaries of an evidence base, and (b) use Bayesian statistical methods to synthesize prior information with estimates of association and summaries of an evidence base to reduce random variability. The premise of the proposed project is that the use of these informatics approaches will reduce the potential for systematic and random error to misleadingly portray research as poorly reproducible and will identify the most important limitations in an evidence base, which will optimize decisions regarding new data collection. The proposed informatics will be extended and applied in the context of two high profile, controversial topic areas with complex data, which will provide examples applicable to other topic areas. For both topic areas, many sources of potential bias have been identified in the surrounding discourse—as have powerful sources of prior information to temper uncertainty—but their influences on individual estimates of association and summaries of the evidence base have not been fully quantified. Quantitative adjustments for these errors using quantitative bias analysis and Bayesian methods—and for publication bias on meta-analytic summaries—would improve reproducibility. We will then extend and apply web-enabled informatics tools to implement the methods for any topic with a set of heterogeneous study results, allowing stakeholders without advanced analytic skills to tailor the underlying assumptions and see for themselves the impact on the summary results. By achieving our aims, this project will advance the use of research informatics to diminish the reproducibility crisis, help to speed consensus-building for any research topic, and productively channel research resources towards resolving the most influential sources of uncertainty in any topic area.

近年来，科学界和非科学界的利益攸关方对缺乏 科学结果的重现性。这些利益相关者将可再生性危机视为 研究人员和编辑的行为。虽然这些行为可能会对重现性产生影响，但有 为什么对复杂数据的研究应该会得出不同的估计，这是可信的理由。首先，每个人 研究容易受到系统性偏差的影响，包括混淆、选择偏差和测量 错误。这些偏见可能是造成重复性差的主要原因。第二，许多研究表明 被批评为缺乏可复制性的是小的，因此受到很大的随机变异性的影响。在……里面 结合显着性检验产生的选择力量，这些小型研究可能会 高估了效果，进一步造成了重复性差的现象。到目前为止，提出的解决方案 对于被认为是可再生性危机的人来说，他们在很大程度上忽视了这些因素。我们建议(A)使用 基于模拟的定量偏差分析技术以调整系统误差对 关联性估计和证据基础总结，以及(B)使用贝叶斯统计方法 将先验信息与关联性估计和证据基础总结相结合，以减少 随机变异性。拟议项目的前提是，使用这些信息学方法将 减少系统和随机错误误导地将研究描述为重复性差的可能性 并将确定证据基础中最重要的限制，这将优化关于新的 数据收集。拟议的信息学将在两个备受瞩目的背景下扩展和应用， 具有复杂数据的有争议的主题领域，这将提供适用于其他主题领域的例子。为 在这两个主题领域，许多潜在的偏见来源都在周围的话语中被发现--就像 强大的先验信息来源，以缓和不确定性-但它们对个人估计的影响 证据基础的关联和摘要尚未完全量化。量化调整： 使用定量偏差分析和贝叶斯方法的这些误差--以及关于荟萃分析的出版物偏差 摘要--将提高再现性。然后，我们将扩展和应用支持Web的信息学工具来 使用一组不同的研究结果为任何主题实施方法，使利益相关者无需 高级分析技能，以量身定制基本假设，并亲自查看对 汇总结果。通过实现我们的目标，该项目将推进研究信息学的使用，以减少 重现性危机，有助于加速为任何研究主题建立共识，并有效地引导 研究资源，以解决任何主题领域中最具影响力的不确定性来源。

项目成果

Yland et al. Respond to "Heuristics and Wish Bias".

伊兰等人。

• DOI: 10.1093/aje/kwac092
• 发表时间: 2022
• 期刊: American journal of epidemiology
• 影响因子: 5
• 作者: Yland,JenniferJ;Wesselink,AmeliaK;Lash,TimothyL;Fox,MatthewP
• 通讯作者: Fox,MatthewP

SAS and R code for probabilistic quantitative bias analysis for misclassified binary variables and binary unmeasured confounders.

用于错误分类二元变量和二元未测量混杂因素的概率定量偏差分析的 SAS 和 R 代码。

• DOI: 10.1093/ije/dyad053
• 发表时间: 2023
• 期刊: International journal of epidemiology
• 影响因子: 7.7
• 作者: Fox,MatthewP;MacLehose,RichardF;Lash,TimothyL
• 通讯作者: Lash,TimothyL

Application of the Adaptive Validation Substudy Design to Colorectal Cancer Recurrence.

适应性验证子研究设计在结直肠癌复发中的应用。

• DOI: 10.2147/clep.s230314
• 发表时间: 2020
• 期刊: Clinical epidemiology
• 影响因子: 3.9
• 作者: Collin,LindsayJ;Riis,AndersH;MacLehose,RichardF;Ahern,ThomasP;Erichsen,Rune;Thorlacius-Ussing,Ole;Lash,TimothyL
• 通讯作者: Lash,TimothyL

Practical data considerations for the modern epidemiology student.

现代流行病学学生的实用数据注意事项。

• DOI: 10.1016/j.gloepi.2021.100066
• 发表时间: 2021
• 期刊: Global epidemiology
• 作者: Tran,NguyenK;Lash,TimothyL;Goldstein,NealD
• 通讯作者: Goldstein,NealD

Opioid-related Mortality in United States Death Certificate Data: A Quantitative Bias Analysis With Expert Elicitation of Bias Parameters.

美国死亡证明数据中与阿片类药物相关的死亡率：专家提取偏差参数的定量偏差分析。

• DOI: 10.1097/ede.0000000000001600
• 发表时间: 2023
• 期刊: Epidemiology (Cambridge, Mass.)
• 作者: Goldsmith,ElizabethS;Krebs,ErinE;Ramirez,MarizenR;MacLehose,RichardF
• 通讯作者: MacLehose,RichardF