External Validation and Comparison of a General Ward Deterioration Index Between Diversely Different Health Systems.
External Validation and Comparison of a General Ward Deterioration Index Between Diversely Different Health Systems.
复制标题
DOI：
10.1097/ccm.0000000000005837
复制
发表时间：
2023-06-01
影响因子：
8.8
通讯作者：
中科院分区：
文献类型：
--
作者：
在新的临床环境中实施预测分析模型充满了挑战。数据集的变化，如临床实践的差异、新的数据采集设备或电子健康记录（EHR）实施的变化，意味着模型看到的输入数据可能与训练数据存在显著差异。因此，在多个机构验证模型至关重要。在这里，使用回顾性数据，我们展示了如何预测重症监护转移和其他UnforReseen事件（PICTURE），在一个单一的学术医疗中心开发的恶化指数，推广到第二个机构与显着不同的患者人群。PICTURE是为普通病房设计的恶化指数，它使用结构化的EHR数据，如实验室值和生命体征。两个大医院的普通病房，一个是学术医疗中心，另一个是社区医院。该模型之前已经在一个大型学术医疗中心的165，018个普通病房中进行了训练和验证。在这里，我们将这个模型应用于来自一家单独的社区医院的11,083次就诊。没有。研究发现，两所医院在缺失率（9/52的特征差异> 5%）、恶化率（4.5%对2.5%）和种族构成（20%非白人对49%非白人）方面存在显著差异。尽管存在这些差异，但PICTURE的表现是一致的（受试者工作特征曲线下面积[AUROC]，0.870; 95% CI，0.861-0.878），精确-召回曲线下面积（AUPRC，0.298; 95%CI，0.275-0.320）在第一医院; AUROC 0.875（0.851-0.902），AUPRC 0.339（0.281-0.398）在第二医院。AUPRC标准化为2.5%的事件发生率。PICTURE在这两个机构的表现也超过了史诗恶化指数和国家预警评分。这两个机构之间存在重大差异，包括数据的可获得性和人口构成。PICTURE能够识别两家医院（AUROC和AUPRC）中存在恶化风险的普通病房患者，并与现有指标进行了比较。
Implementing a predictive analytic model in a new clinical environment is fraught with challenges. Dataset shifts such as differences in clinical practice, new data acquisition devices, or changes in the electronic health record (EHR) implementation mean that the input data seen by a model can differ significantly from the data it was trained on. Validating models at multiple institutions is therefore critical. Here, using retrospective data, we demonstrate how Predicting Intensive Care Transfers and other UnfoReseen Events (PICTURE), a deterioration index developed at a single academic medical center, generalizes to a second institution with significantly different patient population. PICTURE is a deterioration index designed for the general ward, which uses structured EHR data such as laboratory values and vital signs. The general wards of two large hospitals, one an academic medical center and the other a community hospital. The model has previously been trained and validated on a cohort of 165,018 general ward encounters from a large academic medical center. Here, we apply this model to 11,083 encounters from a separate community hospital. None. The hospitals were found to have significant differences in missingness rates (> 5% difference in 9/52 features), deterioration rate (4.5% vs 2.5%), and racial makeup (20% non-White vs 49% non-White). Despite these differences, PICTURE’s performance was consistent (area under the receiver operating characteristic curve [AUROC], 0.870; 95% CI, 0.861–0.878), area under the precision-recall curve (AUPRC, 0.298; 95% CI, 0.275–0.320) at the first hospital; AUROC 0.875 (0.851–0.902), AUPRC 0.339 (0.281–0.398) at the second. AUPRC was standardized to a 2.5% event rate. PICTURE also outperformed both the Epic Deterioration Index and the National Early Warning Score at both institutions. Important differences were observed between the two institutions, including data availability and demographic makeup. PICTURE was able to identify general ward patients at risk of deterioration at both hospitals with consistent performance (AUROC and AUPRC) and compared favorably to existing metrics.