CLINICALLY RELEVANT USE OF PARTIAL ROC CURVES IN PERFORMANCE COMPARISON STUDIES

部分 ROC 曲线在性能比较研究中的临床相关应用

• 批准号: 8155056
• 负责人: DAVID GUR
• 金额: $ 25.91万
• 依托单位: UNIVERSITY OF PITTSBURGH AT PITTSBURGH
• 依托单位国家: 美国
• 财政年份: 2011
• 资助国家: 美国
• 起止时间: 2011-07-15 至 2014-04-30
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/8155056
• 关键词: Address; Area; Characteristics; Clinical; Data; Data Set; Development; Diagnostic; Diagnostic Procedure; Environment; Future; Guidelines; Investigation; Laboratories; Lead; Methodology; Methods; Modality; Performance; Positioning Attribute; Procedures; Publishing; ROC Curve; Receiver Operating Characteristics; Relative (related person); Research; Research Design; Research Personnel; Research Proposals; Sample Size; Specificity; System; Techniques; Testing; Time; base; clinical practice; clinically relevant; design; direct application; experience; improved; indexing; interest; novel strategies; research study; simulation; tool

DESCRIPTION (provided by applicant): The Receiver Operating Characteristic curve (ROC) is the most commonly used approach for assessing and comparing performance levels of diagnostic procedures that may involve the observer as an integral part of the system / practice. One of the more clinically relevant methods of ROC analysis is based on the part of the ROC curve which describes operating characteristics that are expected to be actually observable in clinical practice. A well known example of such an approach is the analysis based on partial area under the ROC curve (partial AUC) over the range of operating points with low false positive fraction (FPF). Although analyses based on a part of the ROC curve are frequently considered to be more relevant, these are not commonly employed due to several important issues. Two of these are the difficulty with a consistent definition of the actual range of interest and the potential loss of efficiency, namely higher likelihood of inconclusive results of a study or the need for a larger sample size for a future study. Our research proposal is primarily related to: 1) Defining the appropriate portion of interest of an ROC curve when estimated for a laboratory experiment performed outside of the actual clinical environment (given that the region of interest is known for the clinical ROC curve). Indeed the same decision threshold in a retrospective laboratory experiment is likely to have different operating characteristics (i.e. FPF and TPF) and therefore the direct application of the range of interest defined in terms of FPF is inappropriate and could lead to inconclusive and less, or totally irrelevant, results; 2) Empirical operating points in performance assessment studies typically span only the region of low FPF and therefore consideration of the entire ROC curve rather than the empirically supported part of the curve may not actually increase the statistical efficiency of comparisons of diagnostic modalities. The research we propose herein will help researchers use partial ROC analysis in a more relevant manner leading to more conclusive results, as well as provide guidelines on the potential statistical tradeoffs between partial and full ROC analyses in commonly encountered experimentally ascertained data, thereby addressing concerns about efficiency. Our proposed effort will provide researchers with three difference methods to define the clinically relevant portion on a laboratory-estimated ROC curve. We will also develop an approach for comparison of partial AUCs by combining strengths of both parametric and nonparametric approaches. This will provide a more interpretable and statistically efficient technique for this purpose. Last, we will investigate the relative efficiency of the partial and full AUC in different types of data commonly encountered experimentally where the empirical operating points span only the range of low FPF, and develop an approach that combines practical relevance of the partial AUC with statistical stability of the full AUC. All investigated analyses will be evaluated using extensive simulations, as well as an assessment of the possible effects, if any, of the proposed analyses on conclusions of previously performed and published large ROC type studies. PUBLIC HEALTH RELEVANCE: The design and analysis of performance assessment studies based on partial ROC curves is frequently more relevant than comparisons of complete ROC curves. However, many analytical and practical issues need to be resolved if a partial ROC approach is to be commonly used for this purpose. We propose to investigate a number of the more important issues in this regard and to develop new tools that will enable clinically relevant use of partial ROC analyses for this purpose and at the same time increase the possibility of conclusive results while decreasing sample size requirements for partial ROC type study designs.

描述(由申请人提供)：受试者工作特性曲线(ROC)是评估和比较诊断程序性能水平的最常用方法，该方法可能涉及观察员作为系统/实践的组成部分。临床上比较相关的ROC分析方法之一是基于ROC曲线的一部分，该曲线描述了预期在临床实践中实际可观察到的操作特征。这种方法的一个众所周知的例子是基于ROC曲线下的部分面积(部分AUC)在具有低假阳性分数(FPF)的工作点范围内的分析。尽管基于部分ROC曲线的分析通常被认为更具相关性，但由于几个重要问题，这些分析并不常用。其中两个问题是对实际感兴趣的范围和潜在效率损失的一致定义的困难，即研究结果不确定的可能性更高，或者需要为未来的研究提供更大的样本量。我们的研究建议主要涉及：1)确定在实际临床环境之外执行的实验室实验估计ROC曲线的适当感兴趣部分(给定感兴趣区域对于临床ROC曲线是已知的)。事实上，在回溯性实验室实验中，相同的决策阈值可能具有不同的操作特性(即，FPF和TPF)，因此直接应用以FPF定义的兴趣范围是不合适的，并且可能导致不确定且较少或完全不相关的结果；2)绩效评估研究中的经验操作点通常只跨越低FPF区域，因此考虑整个ROC曲线而不是曲线的经验性支持部分实际上可能不会增加诊断模式比较的统计效率。我们在此提出的研究将帮助研究人员以更相关的方式使用部分ROC分析，从而获得更具结论性的结果，并就经常遇到的实验确定的数据中部分ROC分析和完全ROC分析之间的潜在统计权衡提供指南，从而解决对效率的担忧。我们提出的努力将为研究人员提供三种不同的方法来定义实验室估计的ROC曲线上的临床相关部分。我们还将开发一种结合参数方法和非参数方法的优点来比较部分AUC的方法。这将为这一目的提供一种更易于解释和统计上更有效的技术。最后，我们将研究部分AUC和完全AUC在实验中常见的不同类型数据中的相对效率，其中经验操作点仅跨越低FPF范围，并开发一种将部分AUC的实际相关性与完整AUC的统计稳定性相结合的方法。所有调查的分析都将使用广泛的模拟进行评估，以及评估建议的分析对先前进行和发表的大型ROC类型研究的结论的可能影响(如果有的话)。 公共卫生相关性：基于部分ROC曲线的绩效评估研究的设计和分析往往比完整ROC曲线的比较更相关。然而，如果要将部分ROC方法普遍用于这一目的，则需要解决许多分析和实践问题。我们建议在这方面调查一些更重要的问题，并开发新的工具，使部分ROC分析在临床上能够用于这一目的，同时增加得出结论的可能性，同时减少部分ROC类型研究设计的样本量要求。