Understanding increments in model performance metrics
Understanding increments in model performance metrics
复制标题
DOI：
10.1007/s10985-012-9238-0
复制
发表时间：
2013-04-01
影响因子：
1.3
通讯作者：
Massaro, Joseph M.
中科院分区：
文献类型：
--
作者：
Pencina, Michael J.;D'Agostino, Ralph B.;Massaro, Joseph M.
接受者工作特征曲线下的面积（AUC）是二元结果预测模型中最常用的判别指标。然而，最近有人批评，当重要的风险因素被添加到具有良好辨别能力的基线模型中时，它无法增加。这导致了一种说法，即依赖AUC作为歧视的衡量标准，可能会错过来自基线模型的风险预测规则在临床表现方面的重要改进。在本文中，我们通过在多元正态性假设下将AUC与临床表现的敏感性和特异性相关联来研究这一说法。对比了识别斜率和AUC的变化规律。我们表明，除非期望有非常好的特异性规则，否则AUC的变化可以充分预测临床表现的变化。然而，需要更强或更多的预测器来实现具有良好和较差判别的基线模型的AUC的相同增量。当需要极好的特异性时，我们的结果表明，区分斜率可能是比AUC更好的模型改进度量。理论结果用一个预测10年房颤发病率模型的弗雷明汉心脏研究实例来说明。
The area under the receiver operating characteristic curve (AUC) is the most commonly reported measure of discrimination for prediction models with binary outcomes. However, recently it has been criticized for its inability to increase when important risk factors are added to a baseline model with good discrimination. This has led to the claim that the reliance on the AUC as a measure of discrimination may miss important improvements in clinical performance of risk prediction rules derived from a baseline model. In this paper we investigate this claim by relating the AUC to measures of clinical performance based on sensitivity and specificity under the assumption of multivariate normality. The behavior of the AUC is contrasted with that of discrimination slope. We show that unless rules with very good specificity are desired, the change in the AUC does an adequate job as a predictor of the change in measures of clinical performance. However, stronger or more numerous predictors are needed to achieve the same increment in the AUC for baseline models with good versus poor discrimination. When excellent specificity is desired, our results suggest that the discrimination slope might be a better measure of model improvement than AUC. The theoretical results are illustrated using a Framingham Heart Study example of a model for predicting the 10-year incidence of atrial fibrillation.