Testing for improvement in prediction model performance.
Testing for improvement in prediction model performance.
复制标题
DOI：
10.1002/sim.5727
复制
发表时间：
2013-04-30
影响因子：
2
通讯作者：
Wang, Zheyu
中科院分区：
文献类型：
--
作者：
Pepe, Margaret Sullivan;Kerr, Kathleen F.;Longton, Gary;Wang, Zheyu
关键词：
近年来，人们提出了新的方法来评估通过在包含一组基线预测因子X的二元结果D的风险模型中添加新的预测因子Y来提高预测性能所获得的改进。我们从理论上证明，关于性能没有改善的零假设等同于在控制X时Y不是风险因素的简单零假设，H0: P (D = 1|X， Y) = P （D = 1|X）。因此，如果Y已经被证明是一个风险因素，那么测试预测性能的改进是多余的。我们还通过模拟研究来研究测试的性质，重点关注ROC曲线下面积（AUC）的变化。一个意想不到的发现是，不调整估计回归系数的可变性的标准测试程序是极其保守的。这可以解释为什么AUC被广泛认为对预测性能的改进不敏感，并表明不敏感的问题与使用无效的推理程序有关，而不是与测量本身有关。为了避免重复测试和使用可能有问题的推断方法，我们建议对没有改进的假设测试仅限于评估Y作为风险因素，这方面的方法已经得到了很好的发展和广泛的应用。对预测性能度量的分析应该集中在评估上，而不是在性能没有改进的情况下进行测试。
New methodology has been proposed in recent years for evaluating the improvement in prediction performance gained by adding a new predictor, Y, to a risk model containing a set of baseline predictors, X, for a binary outcome D. We prove theoretically that null hypotheses concerning no improvement in performance are equivalent to the simple null hypothesis that Y is not a risk factor when controlling for X, H0: P (D = 1|X, Y) = P (D = 1|X). Therefore, testing for improvement in prediction performance is redundant if Y has already been shown to be a risk factor. We also investigate properties of tests through simulation studies, focusing on the change in the area under the ROC curve (AUC). An unexpected finding is that standard testing procedures that do not adjust for variability in estimated regression coefficients are extremely conservative. This may explain why the AUC is widely considered insensitive to improvements in prediction performance and suggests that the problem of insensitivity has to do with use of invalid procedures for inference rather than with the measure itself. To avoid redundant testing and use of potentially problematic methods for inference, we recommend that hypothesis testing for no improvement be limited to evaluation of Y as a risk factor, for which methods are well developed and widely available. Analyses of measures of prediction performance should focus on estimation rather than on testing for no improvement in performance.
登录
查看更多内容
衡量心血管风险的个体预测因素的影响的进步：重新分类措施的作用。
影响因子：
39.2
作者：
Cook NR;Ridker PM
通讯作者：
Ridker PM
影响因子：
1.7
作者：
Cook, Nancy R.;Paynter, Nina P.
通讯作者：
Paynter, Nina P.
影响因子：
2
作者：
Pencina, Michael J.;D'Agostino, Ralph B., Sr.;Steyerberg, Ewout W.
通讯作者：
Steyerberg, Ewout W.
影响因子：
1.9
作者：
Huang, Ying;Pepe, Margaret Sullivan;Feng, Ziding
通讯作者：
Feng, Ziding
DOI：
10.1098/rsta.1933.0009
发表时间：
1933-03-01
影响因子：
--
作者：
Neyman, J
通讯作者：
Neyman, J