Limitations of receiver operating characteristic curve on imbalanced data: Assist device mortality risk scores.
Limitations of receiver operating characteristic curve on imbalanced data: Assist device mortality risk scores.
复制标题
DOI：
10.1016/j.jtcvs.2021.07.041
复制
发表时间：
2023-04
期刊：
影响因子：
--
通讯作者：
Antaki JF
中科院分区：
文献类型：
--
作者：
Movahedi F;Padman R;Antaki JF
在LVAD域中，接收器操作特性（ROC）是分类器的性能的常用度量。然而，ROC可以提供分类器预测短期死亡率的能力的扭曲视图，这是由于存活的患者的比例压倒性地更大，即不平衡的数据。本研究说明了ROC在评估两个分类器90天LVAD死亡率时的模糊性，并引入了精确召回曲线（PRC）作为补充指标，该指标更能代表LVAD分类器在预测少数类别中的作用。本研究针对INTERMACS中记录的800名在2006年至2016年间接受连续流LVAD的患者（测试组）（平均年龄59岁; 146名女性与654名男性），比较了90天LVAD死亡率的两个分类器HeartMate风险评分（HMRS）和随机森林（RF）的ROC和PRC，其中90天死亡率仅为8%。ROC表明RF和HMRS分类器在曲线下面积（AUC）方面的性能相似，分别为0.77和0.63。这与他们的PRC形成对比，RF和HMRS的AUC分别为0.43和0.16。HMRS的PRC显示精密度迅速下降至仅10%，灵敏度略有增加。ROC可以描述分类器或风险评分在应用于不平衡数据时的过度乐观性能。PRC通过关注少数类提供了关于分类器性能的更好的见解。
In the LVAD domain, the receiver operating characteristic (ROC) is a commonly applied metric of performance of classifiers. However, ROC can provide a distorted view of classifiers ability to predict short-term mortality due to the overwhelmingly greater proportion of patients who survive, i.e. imbalanced data. This study illustrates the ambiguity of ROC in evaluating two classifiers of 90-day LVAD mortality and introduces the precision recall curve (PRC) as a supplemental metric that is more representative of LVAD classifiers in predicting the minority class. This study compared the ROC and PRC for two classifiers for 90-day LVAD mortality, HeartMate Risk Score (HMRS) and a Random Forest (RF), for 800 patients (test group) recorded in INTERMACS who received a continuous-flow LVAD between 2006 and 2016 (mean age of 59 years; 146 females vs. 654 males) in which 90-day mortality rate is only 8%. The ROC indicates similar performance of RF and HMRS classifiers with respect to Area Under the Curve (AUC) of 0.77 vs. 0.63, respectively. This is in contrast with their PRC with AUC of 0.43 vs. 0.16 for RF and HMRS, respectively. The PRC for HMRS showed the precision rapidly dropped to only 10% with slightly increasing sensitivity. The ROC can portray an overly-optimistic performance of a classifier or risk score when applied to imbalanced data. The PRC provides better insight about the performance of a classifier by focusing on the minority class.
登录
查看更多内容
影响因子：
8
作者：
O'Brien, Robert;Ishwaran, Hemant
通讯作者：
Ishwaran, Hemant
影响因子：
3.9
作者：
Zhang, Liyuan;Yang, Huamin;Jiang, Zhengang
通讯作者：
Jiang, Zhengang
基于不平衡临床数据预测乳腺癌患者新辅助化疗的病理反应
影响因子：
--
作者：
Gao, Ting;Hao, Yaguang;Han, Bing
通讯作者：
Han, Bing
影响因子：
24
作者：
Cowger, Jennifer;Sundareswaran, Kartik;Slaughter, Mark S.
通讯作者：
Slaughter, Mark S.
影响因子：
7.8
作者：
Mazurowski, Maciej A.;Habas, Piotr A.;Tourassi, Georgia D.
通讯作者：
Tourassi, Georgia D.