Assessing the Impact of Type I and Type II Errors on High Strength of Evidence Outcomes

项目摘要/摘要 说明：请参阅说明。它必须包含适合向公众传播的拟议活动的摘要(否 专有/机密信息)。它应该是对项目的独立描述，并包含目标和方法的声明，以 被雇佣。它应该为在相同或相关领域工作的其他人提供信息。不要超过所提供的空间。 背景：在过去的十年里，对证据强度(SOE)进行评级的系统已经成为 在综合的证据中传达确定性和不确定性的常用工具 系统评价。对于决策者来说，国企评级至关重要，因为它们可以预测未来的影响 对观察到的效果估计的研究。 在最近的一个方法项目中，我们确定国有企业等级的预测价值很低。具体来说， 随着新的研究，超过20%的结果评级为高国企的企业在影响程度上发生了实质性变化 都已经出版了。因为“高SOE”，顾名思义，传达了对正确的强烈信心 估计，如此高比例的变化估计引发了对临床和政策决策的担忧 制作。到目前为止，国企品位预测值低的原因还没有确定。 具体目标：我们提议的研究项目的目标是确定导致 国企等级的预测价值有限。我们建议的研究的具体目标是： 1.确定高SOE结局与I型或II型错误风险增加的比例。 2.确定高SOE结果中I型或II型错误的风险增加是否可以归因于 到不适当的评级(即，给定的结果不应根据当前的情况被评级为高SOE 或者是概念方法中没有发现统计错误风险增加的缺陷。 方法：我们将从Cochrane和医疗保健研究机构抽样100个高国企成果 和质量(AHRQ)报告。我们将重点关注基于以下两种结果的结果(已评级的高SOE 关于随机对照试验的Meta分析，其中数据可用于复制Meta-Rate 分析。为了确定第一类和第二类错误的风险，我们将应用试验序贯分析(TSA) 以确定这些证据中第一类和第二类错误的风险。在SOE较高的情况下 没有得到TSA结果的支持，我们将确定不一致的原因。 影响：我们的结果将提供进一步的洞察，即国有企业有限的预测价值 成绩可以通过更好的指导和培训来提高，或者是国有企业的概念框架 需要进行实质性的反思和修改，如果是这样的话，以什么方式进行修改。

PROJECT SUMMARY/ABSTRACT DESCRIPTION: See instructions. This must contain a summary of the proposed activity suitable for dissemination to the public (no proprietary/confidential information). It should be a self-contained description of the project and contain a statement of objectives and methods to be employed. It should be informative to other persons working in the same or related fields. DO NOT EXCEED THE SPACE PROVIDED. Background: Over the past decade, systems to grade the strength of evidence (SOE) have become commonly used tools to convey certainties and uncertainties in bodies of evidence synthesized in systematic reviews. For decision makers, SOE grades are crucial because they predict the impact of future research on observed estimates of effect. In a recent methods project, we determined that the predictive value of SOE grades is low. Specifically, more than 20% of outcomes graded high SOE substantially changed in magnitude of effect as new studies were published. Because “high SOE,” by definition, conveys strong confidence in the correctness of estimates, such a high proportion of changing estimates raises concerns for clinical and policy decision making. To date, the reasons for the low predictive value of SOE grades have not been determined. Specific Aims: The goal of our proposed research project is to identify factors that are responsible for the limited predictive value of SOE grades. The specific aims of our proposed research are: 1. To determine the proportion of high SOE outcomes with an increased risk of type I or type II errors. 2. To determine whether increased risks of type I or type II errors in high SOE outcomes can be attributed to inappropriate grading (i.e., a given outcome should not have been graded high SOE following current guidance) or to flaws in the conceptual approach that did not detect increased risks for statistical errors. Methods: We will sample 100 high-SOE outcomes from Cochrane and Agency for Healthcare Research and Quality (AHRQ) reports. We will focus on dichotomous outcomes (graded high SOE) that were based on meta-analyses of randomized controlled trails and for which data are available to reproduce the meta- analyses. To determine the risk for type I and type II errors, we will apply Trial Sequential Analyses (TSA) to determine the risks of type I and type II errors in these bodies of evidence. In cases where high SOE is not supported by TSA results, we will determine reasons for the lack of concordance. Implications: Our results will provide further insight as to whether the limited predictive value of SOE grades can be improved by better guidance and training or whether the conceptual framework for SOE needs to be substantially rethought and revised and, if so, in what ways.