Generation & Evaluation of Confidence Bounds for ROC Curves

一代

• 批准号: 0329135
• 负责人: Foster Provost
• 金额: $ 6.71万
• 依托单位: New York University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2003
• 资助国家: 美国
• 起止时间: 2003-08-15 至 2004-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0329135&HistoricalAwards=false
• 关键词: Generation; & Evaluation; Confidence; Bounds

This project develops a general methodology for generating and evaluating confidence intervals/bands for Receiver Operating Characteristic (ROC) curves, a common tool for comparing classification models. Usually two or more ROC curves are compared in one of three ways: by simple visual inspection without confidence assessments, by focusing on one particular point of the ROC curve and generating confidence intervals around that point, or by comparing the areas under the curves. Little work has studied the soundness of ROC confidence intervals, or their use for comparing entire curves. This project consists of six main activities: (1) surveying existing techniques for generating confidence intervals/bands for ROC curves, (2) creating new evaluation metrics that are more appropriate for confidence bands, (3) developing a general framework for generating and evaluating intervals/bands, (4) developing new techniques for creating and optimizing bands, (5) creating a suite of benchmark problems, and (6) evaluating the intervals/bands through large-scale empirical studies to analyze and to characterize their expected containment.The work will lead to a new suite of benchmarks for these types of problems, as well as a general framework for evaluating confidence bands for ROC curves, enabling future researchers to perform similar studies in a well-understood setting. More importantly, the knowledge and techniques that this project produces will affect the many fields, such as medicine, that regularly use ROC analysis. Finally, this project will produce open-source toolkits, so that ROC studies will become much easier.

该项目开发了一种通用方法，用于生成和评估受试者操作特征（ROC）曲线的置信区间/带，这是比较分类模型的常用工具。 通常，两个或多个ROC曲线以三种方式之一进行比较：通过简单的目视检查而不进行置信度评估，通过关注ROC曲线的一个特定点并生成该点周围的置信区间，或者通过比较曲线下面积。 很少有工作研究ROC置信区间的可靠性，或它们用于比较整个曲线。该项目包括六项主要活动：（1）调查用于生成ROC曲线的置信区间/带的现有技术，（2）创建更适合于置信带的新的评估度量，（3）开发用于生成和评估区间/带的通用框架，（4）开发用于创建和优化带的新技术，（5）创建一套基准问题，（6）通过大规模的实证研究来评估区间/带，分析和表征其预期的包容性。这项工作将为这类问题提供一套新的基准，以及评估ROC曲线置信带的一般框架，使未来的研究人员能够在一个良好的理解环境中进行类似的研究。 更重要的是，该项目产生的知识和技术将影响许多经常使用ROC分析的领域，例如医学。 最后，这个项目将产生开放源代码的工具包，这样ROC研究将变得更加容易。