Revisiting performance metrics for prediction with rare outcomes.
Revisiting performance metrics for prediction with rare outcomes.
复制标题
DOI：
10.1177/09622802211038754
复制
发表时间：
2021-10
影响因子：
2.3
通讯作者：
Rose S
中科院分区：
文献类型：
--
作者：
Adhikari S;Normand SL;Bloom J;Shahian D;Rose S
Machine learning algorithms are increasingly used in the clinical literature, claiming advantages over logistic regression. However, they are generally designed to maximize area under the receiver operating characteristic curve (AUC). While AUC and other measures of accuracy are commonly reported for evaluating binary prediction problems, these metrics can be misleading. We aim to give clinical and machine learning researchers a realistic medical example of the dangers in relying on a single measure of discriminatory performance to evaluate binary prediction questions. Prediction of medical complications after surgery is a frequent but challenging task because many post-surgery outcomes are rare. We predicted post-surgery mortality among patients in a clinical registry who received at least one aortic valve replacement. Estimation incorporated multiple evaluation metrics and algorithms typically regarded as performing well with rare outcomes, as well as an ensemble and a new extension of the lasso for multiple unordered treatments. Results demonstrated high accuracy for all algorithms with moderate measures of cross-validated AUC. False positive rates were less than 1%, however, true positive rates were less than 7%, even when when paired with a 100% positive predictive value, and graphical representations of calibration were poor. Similar results were seen in simulations, with the addition of high AUC (>90%) accompanying low true positive rates. Clinical studies should not primarily report AUC or accuracy.
登录
查看更多内容
影响因子：
2
作者：
Heinze, G;Schemper, M
通讯作者：
Schemper, M
影响因子：
37.8
作者：
Cook, Nancy R.
通讯作者：
Cook, Nancy R.
影响因子：
158.5
作者：
Goldstone, Andrew B.;Chiu, Peter;Woo, Y. Joseph
通讯作者：
Woo, Y. Joseph
影响因子：
1.7
作者：
Austin, Peter C.;Lee, Douglas S.;Tu, Jack V.
通讯作者：
Tu, Jack V.
影响因子：
1.8
作者：
Friedman, JH
通讯作者：
Friedman, JH