Assessing the Impact of Type I and Type II Errors on High Strength of Evidence Outcomes

评估 I 类和 II 类错误对高强度证据结果的影响

• 批准号: 9164240
• 负责人: Gerald Gartlehner
• 金额: $ 9.93万
• 依托单位: RESEARCH TRIANGLE INSTITUTE
• 依托单位国家: 美国
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-09-02 至 2018-11-30
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/9164240
• 关键词: Assessing; Impact; Type; II; Errors

PROJECT SUMMARY/ABSTRACT DESCRIPTION: See instructions. This must contain a summary of the proposed activity suitable for dissemination to the public (no proprietary/confidential information). It should be a self-contained description of the project and contain a statement of objectives and methods to be employed. It should be informative to other persons working in the same or related fields. DO NOT EXCEED THE SPACE PROVIDED. Background: Over the past decade, systems to grade the strength of evidence (SOE) have become commonly used tools to convey certainties and uncertainties in bodies of evidence synthesized in systematic reviews. For decision makers, SOE grades are crucial because they predict the impact of future research on observed estimates of effect. In a recent methods project, we determined that the predictive value of SOE grades is low. Specifically, more than 20% of outcomes graded high SOE substantially changed in magnitude of effect as new studies were published. Because “high SOE,” by definition, conveys strong confidence in the correctness of estimates, such a high proportion of changing estimates raises concerns for clinical and policy decision making. To date, the reasons for the low predictive value of SOE grades have not been determined. Specific Aims: The goal of our proposed research project is to identify factors that are responsible for the limited predictive value of SOE grades. The specific aims of our proposed research are: 1. To determine the proportion of high SOE outcomes with an increased risk of type I or type II errors. 2. To determine whether increased risks of type I or type II errors in high SOE outcomes can be attributed to inappropriate grading (i.e., a given outcome should not have been graded high SOE following current guidance) or to flaws in the conceptual approach that did not detect increased risks for statistical errors. Methods: We will sample 100 high-SOE outcomes from Cochrane and Agency for Healthcare Research and Quality (AHRQ) reports. We will focus on dichotomous outcomes (graded high SOE) that were based on meta-analyses of randomized controlled trails and for which data are available to reproduce the meta- analyses. To determine the risk for type I and type II errors, we will apply Trial Sequential Analyses (TSA) to determine the risks of type I and type II errors in these bodies of evidence. In cases where high SOE is not supported by TSA results, we will determine reasons for the lack of concordance. Implications: Our results will provide further insight as to whether the limited predictive value of SOE grades can be improved by better guidance and training or whether the conceptual framework for SOE needs to be substantially rethought and revised and, if so, in what ways.

项目总结/摘要 描述：参见说明。这必须包含一个适合向公众传播的拟议活动的摘要（没有 专有/机密信息）。它应该是一个独立的项目描述，并包含一个目标和方法的声明， 被雇用。它应该对在相同或相关领域工作的其他人提供信息。不要超过所提供的空间。 背景：在过去的十年中，证据强度（SOE）分级系统已成为 常用的工具来传达综合证据中的不确定性和不确定性， 系统评价对于决策者来说，SOE等级至关重要，因为它们可以预测未来的影响。 研究观察到的效果估计。 在最近的一个方法项目中，我们确定SOE等级的预测值很低。具体地说， 超过20%的结局分级为高SOE， 发表了。因为“高SOE”，顾名思义，传达了对正确性的强烈信心， 估计，如此高比例的变化估计引起了临床和政策决策的关注 制作。到目前为止，SOE等级预测值低的原因尚未确定。 具体目标：我们提出的研究项目的目标是确定负责的因素 SOE等级的预测价值有限。我们建议的研究的具体目标是： 1.确定I型或II型错误风险增加的高SOE结局的比例。 2.确定高SOE结局中I型或II型错误的风险增加是否可归因于 不适当的分级（即，在当前情况下，给定结局不应被评为高SOE 这可能是由于统计方法的缺陷（即没有发现统计错误风险的增加）。 方法：我们将从科克伦和卫生保健研究机构中抽取100个高SOE结果 质量（AHRQ）报告我们将重点关注基于以下因素的二分结局（分级的高SOE）： 对随机对照试验的荟萃分析，并且数据可用于重现Meta分析， 分析。为了确定I型和II型错误的风险，我们将应用试验序贯分析（TSA） 以确定这些证据中第一类和第二类错误的风险。如果高SOE是 如果TSA结果不支持，我们将确定缺乏一致性的原因。 影响：我们的研究结果将提供进一步的见解，是否有限的预测价值的国有企业 可以通过更好的指导和培训或SOE的概念框架来提高成绩 需要进行实质性的重新思考和修订，如果需要，以何种方式进行。