ROC Analysis of Imaging System Evaluations

描述（由申请人提供）：当评价成像系统是否可能用于临床环境以及监管批准过程中时，技术和实践的诊断性能评估构成决策过程中越来越重要的一部分。在医学成像中，这些评估的很大一部分包括使用ROC响应或其衍生物进行分析的任务特定评级。这个项目的总体目标是提高我们的实际（方法）和计算（分析相关）的问题，我们遇到的系统评估和比较时，观察员，计算机化的计划（如CAD），或两者的组合成为诊断系统的一个组成部分，我们的理解。在当前的资助期间，我们开发并测试了新的方法来分析ROC研究期间确定的评分数据，并开始解决与将基于相关性的权重纳入分析相关的问题。我们正在探索的基本方法使用阴性和阳性病例（检查）之间的所有配对观察结果，使用非参数方法构建性能的汇总指数。无条件和有条件的置换为基础的测试已被开发和测试，并被证明有明显的实验优势，在几个实验设置，特别是小样本量的研究。六个主要的努力，建议使用这种一般方法。我们建议将该方法扩展到多读者，多模态，多疾病设置，以及将各种效用函数的分析。我们还将研究如何使用这种方法和其他方法来识别并可能处理ROC类型研究中的离群值。最后，使用相同的基本概念，我们建议扩大调查和研究的理想自助方差的非参数估计的AUC的其他现有的估计在多读者环境中的关系。我们将开发封闭形式的解决方案进行方差分析程序对所有可能的自助值。

DESCRIPTION (provided by applicant): Assessment of diagnostic performance of technologies and practices constitute an increasingly important part of the decision making process when imaging systems are evaluated for possible use in the clinical environment as well as during the regulatory approval process. In medical imaging a substantial fraction of these evaluations include task specific ratings that are being analyzed using an ROC response or derivative thereof. The overall objective of this project is to improve our understanding of the practical (methodological) and computational (analyses related) issues that we encounter during systems evaluations and comparisons when the observer, a computerized scheme (e.g. CAD), or a combination of both become an integral part of the diagnostic system. During the current period of funding, we developed and tested new approaches to analyzing scoring data ascertained during the performance of ROC studies and began to address issues related to the incorporation of relevance based weights into the analyses. The underlying approach we are exploring uses all paired observations between negative and positive cases (examinations) to construct summary indices of performance using a non-parametric methodology. Both unconditional and conditional permutation based tests have been developed and tested and were shown to have distinct experimental advantages in several experimental settings and in particular, for small sample size studies. Six primary efforts are proposed using this general approach. We propose to expand the method to a multi-reader, multimodality, multi-disease settings as well as to incorporate a variety of utility functions into the analysis. We will also investigate how this and other methods can be used to identify and possibly treat outliers in ROC type studies. Last, using the same underlying concept we propose to expand the investigations and study the relationship of the ideal bootstrap variances of the non-parametric estimators of the AUC to other existing estimators in the multi-reader environment. We will develop closed-form solutions for the ANOVA procedure conducted on all possible bootstrap values.