Validation and Statistical Power Comparison of Methods for Analyzing Free-response Observer Performance Studies
Validation and Statistical Power Comparison of Methods for Analyzing Free-response Observer Performance Studies
复制标题
DOI：
10.1016/j.acra.2008.07.018
复制
发表时间：
2008-12-01
影响因子：
4.8
通讯作者：
Chakraborty, Dev P.
中科院分区：
文献类型：
--
作者：
Chakraborty, Dev P.
基本原理和目标。这项工作的目的是验证和比较所提出的使用基于搜索模型的模拟器分析自由响应数据的方法的统计功效。材料和方法。描述了一种自由响应数据模拟器，它可以模拟单个阅读器以两种模式解释相同的情况，或两个计算机辅助检测（CAD）算法，或两个人类观察者以一种模式解释相同的情况。描述了类似于 Roe 和 Metz 接收器操作特性 (ROC) 数据模拟器的方差分量模型；它模拟自由反应研究中的案例内和多式联运相关性。模拟了两个通用观察者：一个准人类观察者和一个准 CAD 算法。研究了 ROC、折刀替代自由响应操作特征 (JAFROC)、称为 JAFROC-1 的 JAFROC 变体、初始检测和候选分析 (IDCA) 以及非参数 (NP) 方法的原假设 (NH) 有效性和统计功效。结果。所有方法在各种模拟器参数上都具有有效的 NH 行为。对于相同数量的正常和异常病例，对于人类观察者来说，方法的统计功效排序为 JAFROC-1 > JAFROC > (IDCA 类似于 NP) > ROC。对于CAD算法，排名为（NP类似于IDCA）>（JAFROC-1类似于JAFROC）>ROC。无论哪种情况，排名最高的方法的统计功效都超过排名最低的方法的大约两倍。统计功效对模拟器参数的依赖性遵循预期趋势。对于异常情况多于正常情况的数据集，JAFROC-1 功效显着超过 JAFROC 功效。结论。基于这项工作，建议对人类观察者（包括具有 CAD 辅助的人类观察者）使用 JAFROC-1，并使用 NP 方法来评估 CAD 算法。
Rationale and Objectives. The aim of this work was to validate and compare the statistical powers of proposed methods for analyzing free-response data using a search-model-based simulator.Materials and Methods. A free-response data simulator is described that can model a single reader interpreting the same cases in two modalities, or two computer-aided detection (CAD) algorithms, or two human observers, interpreting the same cases in one modality. A variance components model, analogous to the Roe and Metz receiver-operating characteristic (ROC) data simulator, is described; it models intracase and intermodality correlations in free-response studies. Two generic observers were simulated: a quasi-human observer and a quasi-CAD algorithm. Null hypothesis (NH) validity and statistical powers of ROC, jackknife alternative free-response operating characteristic (JAFROC), a variant of JAFROC termed JAFROC-1, initial detection and candidate analysis (IDCA), and a nonparametric (NP) approach were investigated.Results. All methods had valid NH behavior over a wide range of simulator parameters. For equal numbers of normal and abnormal cases, for the human observer, the statistical power ranking of the methods was JAFROC-1 > JAFROC > (IDCA similar to NP) > ROC. For the CAD algorithm, the ranking was (NP similar to IDCA) > (JAFROC-1 similar to JAFROC) > ROC. In either case, the statistical power of the highest ranked method exceeded that of the lowest ranked method by about a factor of two. Dependence of statistical power on simulator parameters followed expected trends. For data sets with more abnormal cases than normal cases, JAFROC-1 power significantly exceeded JAFROC power.Conclusion. Based on this work, the recommendation is to use JAFROC-1 for human observers (including human observers with CAD assist) and the NP method for evaluating CAD algorithms.