Utility-Based Assessment of Diagnostic Imaging Performance

描述（由申请人提供）：观察员性能研究已成为评价诊断成像器械的既定方法，并且通常需要证实出版物和监管机构（如FDA）中的声明。这些研究通常使用受试者工作特征（ROC）分析进行评估，研究旨在推广到读者和病例人群。这些研究的典型终点是ROC曲线下面积（AUC）。然而，AUC忽略了大多数诊断任务集中在ROC曲线的某些区域的事实。例如，在筛查乳房X线摄影中，高于20%的假阳性率（其构成曲线的大部分）在实践中根本不现实。 在这个建议中，我们寻求开发一种替代的品质因数，用于评估观察员的性能研究，是基于预期效用（EU）的概念。欧盟将诊断结果纳入性能指标，因此将分析限制在ROC曲线的最相关区域。初步证据表明，EU可能比AUC具有更大的统计功效，用于检测完全交叉实验中的模态差异，其中所有阅片人对所有成像模态中的所有图像进行评分。我们还寻求扩展欧盟的分析到本地化和自由响应ROC范式，它以前没有被用作其他理论设备。 我们提出的三个具体目标反映了这些发展和传播目标。我们建议（目标1）通过分析导致EU终点最大统计功效的参数，研究ROC研究中EU的最佳实验设计，比较定位和自由应答评估范式中的EU与AUC（目标2），以了解这些方法中的实验设计是否有任何益处，并开发可分发的软件（目标3），使该领域的调查人员能够将欧盟用于自己的数据或自己的评估。

DESCRIPTION (provided by applicant): Observer performance studies have become an established methodology for evaluating diagnostic imaging devices, and are often required for substantiating claims made in publications and to regulatory agencies such as the FDA. These studies are typically evaluated using receiver operating characteristic (ROC) analysis, with studies designed to generalize to the population of readers and cases. The typical endpoint of such studies is the area under the ROC curve (AUC). However, AUC neglects the fact that most diagnostic tasks are focused on certain regions of the ROC curve. For example, in screening mammography false positive rates above 20% (which constitutes most of the curve) are simply not realistic in practice. In this proposal we seek to develop an alternative figure of merit for evaluating observer performance studies that is based on the concept of expected utility (EU). EU incorporates diagnostic outcomes into the performance measure, and thus confines analysis to the most relevant region of the ROC curve. Preliminary evidence suggests that EU may have greater statistical power than AUC for detecting modality differences in fully-crossed experiments in which all readers score all images in all imaging modalities. We also seek to extend the EU analysis into localization and free-response ROC paradigms where it has not previously been used other than as a theoretical device. The three specific aims we propose reflect these goals of development and dissemination. We propose (Aim 1) to investigate optimal experimental design for EU in ROC studies, by analyzing parameters that lead to the greatest statistical power for the EU endpoint, to compare EU to AUC in localization and free-response assessment paradigms (Aim 2) to understand if there is any benefit in experimental design in these methods, and to develop distributable software (Aim 3) that allows investigators in the field to use EU on their own data or for their own assessments.