ROC Analysis of Imaging System Evaluations

描述(由申请人提供)：当成像系统被评估可能用于临床环境以及在监管批准过程中时，对技术和实践的诊断性能的评估构成决策过程中越来越重要的部分。在医学成像中，这些评估中的很大一部分包括使用ROC响应或其导数进行分析的特定任务评级。本项目的总体目标是提高我们对当观察者、计算机化方案(例如CAD)或两者的组合成为诊断系统的组成部分时，我们在系统评估和比较期间遇到的实际(方法学)和计算(相关分析)问题的理解。在目前的资助期间，我们开发和测试了新的方法来分析在执行ROC研究期间确定的评分数据，并开始处理与将基于相关性的权重纳入分析有关的问题。我们正在探索的基本方法使用负面和正面案例(检查)之间的所有配对观察，使用非参数方法构建绩效摘要指数。基于无条件和条件置换的测试已经被开发和测试，并且在几个实验设置中被证明具有明显的实验优势，特别是对于小样本量研究。使用这一一般方法提出了六项主要工作。我们建议将该方法扩展到多读者、多模式、多疾病环境，并将各种实用函数纳入分析。我们还将研究如何使用这种方法和其他方法来识别和可能处理ROC类型研究中的离群值。最后，利用相同的基本概念，我们提出了在多读者环境下扩展调查和研究AUC的非参数估计的理想Bootstrap方差与其他已有估计的关系。我们将为对所有可能的Bootstrap值执行的ANOVA过程开发封闭形式的解。

DESCRIPTION (provided by applicant): Assessment of diagnostic performance of technologies and practices constitute an increasingly important part of the decision making process when imaging systems are evaluated for possible use in the clinical environment as well as during the regulatory approval process. In medical imaging a substantial fraction of these evaluations include task specific ratings that are being analyzed using an ROC response or derivative thereof. The overall objective of this project is to improve our understanding of the practical (methodological) and computational (analyses related) issues that we encounter during systems evaluations and comparisons when the observer, a computerized scheme (e.g. CAD), or a combination of both become an integral part of the diagnostic system. During the current period of funding, we developed and tested new approaches to analyzing scoring data ascertained during the performance of ROC studies and began to address issues related to the incorporation of relevance based weights into the analyses. The underlying approach we are exploring uses all paired observations between negative and positive cases (examinations) to construct summary indices of performance using a non-parametric methodology. Both unconditional and conditional permutation based tests have been developed and tested and were shown to have distinct experimental advantages in several experimental settings and in particular, for small sample size studies. Six primary efforts are proposed using this general approach. We propose to expand the method to a multi-reader, multimodality, multi-disease settings as well as to incorporate a variety of utility functions into the analysis. We will also investigate how this and other methods can be used to identify and possibly treat outliers in ROC type studies. Last, using the same underlying concept we propose to expand the investigations and study the relationship of the ideal bootstrap variances of the non-parametric estimators of the AUC to other existing estimators in the multi-reader environment. We will develop closed-form solutions for the ANOVA procedure conducted on all possible bootstrap values.