Utility-Based Assessment of Diagnostic Imaging Performance

描述(由申请人提供)：观察者绩效研究已成为评估诊断成像设备的既定方法，并经常需要证实出版物中的声明和向FDA等监管机构提出的声明。这些研究通常使用接收者操作特征(ROC)分析进行评估，研究旨在将其推广到读者和案例人群。这类研究的典型终点是ROC曲线下面积(AUC)。然而，AUC忽略了这样一个事实，即大多数诊断任务都集中在ROC曲线的某些区域。例如，在筛查乳房X光检查中，假阳性率高于20%(构成曲线的大部分)在实践中是不现实的。在这项提案中，我们寻求开发一个替代的品质因数，用于评价基于预期效用(EU)概念的观察员业绩研究。EU将诊断结果纳入绩效衡量，因此将分析限制在ROC曲线中最相关的区域。初步证据表明，在完全交叉实验中，在所有阅读器对所有成像模式的所有图像进行评分的情况下，EU可能比AUC具有更大的统计能力来检测模式差异。我们还试图将欧盟的分析扩展到本地化和自由反应的ROC范式，在这些范式中，它以前除了作为一种理论工具之外还没有被使用过。我们提出的三个具体目标反映了这些发展和传播目标。我们建议(目标1)在中华民国研究中调查欧盟的最佳实验设计，通过分析导致欧盟终点的最大统计威力的参数，在本地化和自由反应评估范式中比较欧盟和AUC(目标2)，以了解这些方法在实验设计中是否有任何益处，并开发可分发软件(目标3)，允许该领域的研究人员根据自己的数据或自己的评估使用欧盟。

DESCRIPTION (provided by applicant): Observer performance studies have become an established methodology for evaluating diagnostic imaging devices, and are often required for substantiating claims made in publications and to regulatory agencies such as the FDA. These studies are typically evaluated using receiver operating characteristic (ROC) analysis, with studies designed to generalize to the population of readers and cases. The typical endpoint of such studies is the area under the ROC curve (AUC). However, AUC neglects the fact that most diagnostic tasks are focused on certain regions of the ROC curve. For example, in screening mammography false positive rates above 20% (which constitutes most of the curve) are simply not realistic in practice. In this proposal we seek to develop an alternative figure of merit for evaluating observer performance studies that is based on the concept of expected utility (EU). EU incorporates diagnostic outcomes into the performance measure, and thus confines analysis to the most relevant region of the ROC curve. Preliminary evidence suggests that EU may have greater statistical power than AUC for detecting modality differences in fully-crossed experiments in which all readers score all images in all imaging modalities. We also seek to extend the EU analysis into localization and free-response ROC paradigms where it has not previously been used other than as a theoretical device. The three specific aims we propose reflect these goals of development and dissemination. We propose (Aim 1) to investigate optimal experimental design for EU in ROC studies, by analyzing parameters that lead to the greatest statistical power for the EU endpoint, to compare EU to AUC in localization and free-response assessment paradigms (Aim 2) to understand if there is any benefit in experimental design in these methods, and to develop distributable software (Aim 3) that allows investigators in the field to use EU on their own data or for their own assessments.