Observer studies involving detection and localization: Modeling, analysis, and validation
Observer studies involving detection and localization: Modeling, analysis, and validation
复制标题
DOI：
10.1118/1.1769352
复制
发表时间：
2004-08-01
期刊：
影响因子：
3.8
通讯作者：
Berbaum, KS
中科院分区：
文献类型：
--
作者：
Chakraborty, DP;Berbaum, KS
虽然接受者工作特征（ROC）范式是评估诊断成像系统的公认方法，但它有一些严重的缺点，因为它仅限于每个图像一个观察者报告。相比之下，自由反应ROC （FROC）范式和相关分析方法允许观察者在每次成像研究中报告多个异常，并使用报告异常的位置来改进测量。由于ROC方法不能容纳多个响应或使用位置信息，其统计能力将受到影响。由于担心对同一诊断研究的反应是否可以被视为独立的，FROC范式/分析尚未得到广泛接受。本文提出了一种新的不做独立性假设的叠刀法（JAFROC）。新的分析方法结合了FROC和Dorfman -Berbaum-Metz （DBM）方法的元素。为了将JAFROC与早期的自由响应分析方法（特别是替代的自由响应方法，或AFROC方法）以及使用传统ROC评分的DBM方法进行比较，我们开发了一个模型来生成模拟的FROC数据。仿真模型是基于专家如何评估图像的眼动模型。它允许我们检查零假设（NH）行为和不同方法的统计能力。我们发现AFROC分析没有通过NH检验，过于保守。JAFROC方法和DBM方法均通过了NH检验，但JAFROC比DBM方法具有更强的统计能力。这一比较的结果表明，通过使用JAFROC方法，未来的诊断性能研究可能会提高统计能力或减少样本量要求。(C) 2004年美国医学物理学家协会。
Although the receiver operating characteristic (ROC) paradigm is the accepted method for evaluation of diagnostic imaging systems, it has some serious shortcomings inasmuch as it is restricted to one observer report per image. By contrast the free-response ROC (FROC) paradigm and associated analysis method allows the observer to report multiple abnormalities within each imaging study, and uses the location of reported abnormalities to improve the measurement. Because the ROC method cannot accommodate multiple responses or use location information, its statistical power will suffer. The FROC paradigm/analysis has not enjoyed widespread acceptance because of concern about whether responses made to the same diagnostic study can be treated as independent. We propose a new jackknife FROC analysis method (JAFROC) that does not make the independence assumption. The new analysis method combines elements of FROC and the Dorfman -Berbaum-Metz (DBM) methods. To compare JAFROC to an earlier free-response analysis method (specifically the alternative free-response, or AFROC method), and to the DBM method, which uses conventional ROC scoring, we developed a model for generating simulated FROC data. The simulation model is based on an eye-movement model of how experts evaluate images. It allowed us to examine null hypothesis (NH) behavior and statistical power of the different methods. We found that AFROC analysis did not pass the NH test, being unduly conservative. Both the JAFROC method and the DBM method passed the NH test, but JAFROC had more statistical power than the DBM method. The results of this comparison suggest that future studies of diagnostic performance may enjoy improved statistical power or reduced sample size requirements through the use of the JAFROC method. (C) 2004 American Association of Physicists in Medicine.