Prediction Accuracy of a Sample-size Estimation Method for ROC Studies
Prediction Accuracy of a Sample-size Estimation Method for ROC Studies
复制标题
DOI：
10.1016/j.acra.2010.01.007
复制
发表时间：
2010-05-01
影响因子：
4.8
通讯作者：
Chakraborty, Dev P.
中科院分区：
文献类型：
--
作者：
Chakraborty, Dev P.
原理和目的：样本量估计是计划受试者工作特征（ROC）研究时的重要考虑因素。这项工作的目的是评估的预测精度的样本量估计方法，使用Monte Carlo simulation method.Materials和方法：两个ROC评级模拟器，其特征在于低读者和高的情况下的变异性（LH）和高读者和低的情况下的变异性（HL）被用来生成试点数据集在两种模式。Dorfman-Berbaum-梅斯多读者多病例（DBM-MRMC）的评级分析得出了模态读者，模态病例和误差方差的估计。这些数据被输入到Hillis-Berbaum（HB）样本量估计方法中，该方法预测了关键研究中10名阅片员达到80%把握度和0.06效应量所需的病例数。生成了一般化到阅片人和病例（随机-全部）、仅病例（随机-病例）和仅阅片人（随机-阅片人）的预测。一个预测准确度指数定义为任何单一的预测产生真正的权力在75%-90%的范围内的概率被用来评估HB method.Results：随机情况下的泛化，HB方法的预测准确度是合理的，类似于50%的5个读者和100例中的试点研究。LH条件下的预测精度一般高于HL条件下。在理想条件下（许多读者在试点研究中），基于DBM-MRMC的HB方法高估了病例数。当阅片者变异性较大（HL）时，高估可以通过较大的模态-阅片者方差估计来解释。在试点研究中增加读者数量的最大好处是实现LH，其中15个读者足以产生预测准确度>50%，在所有泛化条件下，但好处是较小的HL，其中预测准确度为36%，15个读者在随机所有和随机读者conditions.Conclusion：HB方法往往高估的情况下的数量。随机情况推广具有合理的预测精度。如果在初步研究中使用了约15名读片员，则该方法在LH的所有条件下均可合理执行。当阅片者变异性较大时，所有随机和随机阅片者概括的预测准确性会受到影响。研究设计者可能希望将HB预测与其他方法的预测以及先前类似研究中使用的样本量进行比较。
Rationale and Objectives: Sample-size estimation is an important consideration when planning a receiver operating characteristic (ROC) study. The aim of this work was to assess the prediction accuracy of a sample-size estimation method using the Monte Carlo simulation method.Materials and Methods: Two ROC ratings simulators characterized by low reader and high case variabilities (LH) and high reader and low case variabilities (HL) were used to generate pilot data sets in two modalities. Dorfman-Berbaum-Metz multiple-reader multiple-case (DBM-MRMC) analysis of the ratings yielded estimates of the modality-reader, modality-case, and error variances. These were input to the Hillis-Berbaum (HB) sample-size estimation method, which predicted the number of cases needed to achieve 80% power for 10 readers and an effect size of 0.06 in the pivotal study. Predictions that generalized to readers and cases (random-all), to Cases only (random-cases), and to readers only (random-readers) were generated. A prediction-accuracy index defined as the probability that any single prediction yields true power in the 75%-90% range was used to assess the HB method.Results: For random-case generalization, the HB-method prediction-accuracy was reasonable, similar to 50% for five readers and 100 cases in the pilot study. Prediction-accuracy was generally higher under LH conditions than under HL conditions. Under ideal conditions (many readers in the pilot study) the DBM-MRMC-based HB method overestimated the number of cases. The overestimates could be explained by the larger modality-reader variance estimates when reader variability was large (HL). The largest benefit of increasing the number of readers in the pilot study was realized for LH, where 15 readers were enough to yield prediction accuracy >50% under all generalization conditions, but the benefit was lesser for HL where prediction accuracy was similar to 36% for 15 readers under random-all and random-reader conditions.Conclusion: The HB method tends to overestimate the number of cases. Random-case generalization had reasonable prediction accuracy. Provided about 15 readers were used in the pilot study the method performed reasonably under all conditions for LH. When reader variability was large, the prediction-accuracy for random-all and random-reader generalizations was compromised. Study designers may wish to compare the HB predictions to those of other methods and to sample-sizes used in previous similar studies.