ROC Analysis of Imaging System Evaluations

描述（由申请人提供）：当评估成像系统在临床环境中的可能使用以及在监管批准过程中，技术和实践的诊断性能评估构成决策过程中越来越重要的一部分。在医学成像中，这些评估的很大一部分包括使用ROC反应或其衍生分析的任务特定评级。该项目的总体目标是提高我们对系统评估和比较中遇到的实际（方法学）和计算（分析相关）问题的理解，当观察者、计算机化方案（例如CAD）或两者的组合成为诊断系统的组成部分时。在当前资助期间，我们开发并测试了分析ROC研究绩效期间确定的评分数据的新方法，并开始解决与将基于相关性的权重纳入分析相关的问题。我们正在探索的基本方法是使用负面和正面案例（检查）之间的所有成对观察，使用非参数方法构建绩效的总结指标。已经开发和测试了基于无条件和条件排列的测试，并在若干实验环境中，特别是在小样本量研究中显示出明显的实验优势。采用这种一般方法提出了六项主要工作。我们建议将该方法扩展到多读者、多模态、多疾病环境，并将各种实用函数纳入分析。我们还将研究如何使用这种方法和其他方法来识别和可能处理ROC型研究中的异常值。最后，利用相同的基本概念，我们提出扩展调查和研究在多阅读器环境下AUC的非参数估计量的理想自举方差与其他现有估计量的关系。我们将开发对所有可能的引导值进行方差分析过程的封闭形式的解决方案。

DESCRIPTION (provided by applicant): Assessment of diagnostic performance of technologies and practices constitute an increasingly important part of the decision making process when imaging systems are evaluated for possible use in the clinical environment as well as during the regulatory approval process. In medical imaging a substantial fraction of these evaluations include task specific ratings that are being analyzed using an ROC response or derivative thereof. The overall objective of this project is to improve our understanding of the practical (methodological) and computational (analyses related) issues that we encounter during systems evaluations and comparisons when the observer, a computerized scheme (e.g. CAD), or a combination of both become an integral part of the diagnostic system. During the current period of funding, we developed and tested new approaches to analyzing scoring data ascertained during the performance of ROC studies and began to address issues related to the incorporation of relevance based weights into the analyses. The underlying approach we are exploring uses all paired observations between negative and positive cases (examinations) to construct summary indices of performance using a non-parametric methodology. Both unconditional and conditional permutation based tests have been developed and tested and were shown to have distinct experimental advantages in several experimental settings and in particular, for small sample size studies. Six primary efforts are proposed using this general approach. We propose to expand the method to a multi-reader, multimodality, multi-disease settings as well as to incorporate a variety of utility functions into the analysis. We will also investigate how this and other methods can be used to identify and possibly treat outliers in ROC type studies. Last, using the same underlying concept we propose to expand the investigations and study the relationship of the ideal bootstrap variances of the non-parametric estimators of the AUC to other existing estimators in the multi-reader environment. We will develop closed-form solutions for the ANOVA procedure conducted on all possible bootstrap values.