Opening The Black Box: Enhancing Machine Learning Interpretability To Optimize Clinical Response To Sudden Deterioration In COVID-19 Patients

打开黑匣子：增强机器学习的可解释性，以优化对 COVID-19 患者突然恶化的临床反应

• 批准号: 10259197
• 负责人: Dana Peres Edelson
• 金额: $ 199.6万
• 依托单位: AGILEMD, INC.
• 依托单位国家: 美国
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-09-22 至 2025-03-21
• 项目状态: 未结题
• 来源: https://reporter.nih.gov/project-details/10259197
• 关键词: Acute; Adoption; Adult; Algorithms; COVID-19; COVID-19 patient; Chicago; Clinical; Data Set; Deterioration; Development; Effectiveness; Electronic Health Record; Ensure; Evaluation; Expert Opinion; Goals; Health Personnel; Healthcare; Hemorrhage; Hospitals; Human; Individual; Intervention; Interview; Logic; Logistic Regressions; Machine Learning; Measures; Medical Staff; Modeling; Multicenter Trials; Nature; Patient-Focused Outcomes; Patients; Phase; Predictive Analytics; Process; Provider; Relative Risks; Respiratory Failure; Retrospective cohort; Risk; Risk Reduction; Running; Sepsis; Shock; Small Business Innovation Research Grant; System; Testing; Time; Trust; Universities; User-Computer Interface; Validation; Visualization; Weight; Workload; base; clinical decision support; clinical implementation; clinical practice; design; distrust; experience; hemodynamics; improved; improved outcome; insight; machine learning algorithm; mortality; novel; pandemic disease; predict clinical outcome; prediction algorithm; predictive modeling; prospective; response; satisfaction; septic; simulation; standard care; standard of care; tool; trend; uptake; usability

Project Summary/Abstract Advanced machine learning (ML) has consistently been shown to outperform expert opinion and more simple analytics for predicting clinical outcomes. However, there has been a paucity of successful prospective clinical implementations of such tools. The unique barriers to advanced ML implementation and adoption in healthcare are (1) the technological challenges of running and displaying these models in real-time within existing workflows and (2) a general distrust for black box algorithms among highly skilled providers. As a result, the promise of these tools is largely lost in healthcare. This is particularly problematic in COVID-19, where patients can deteriorate rapidly, from appearing stable to suddenly being in respiratory failure or shock with little obvious warning. Early recognition of this deterioration is vital to proactive interventions, which can improve outcomes. eCART is a predictive analytic that has been developed iteratively at the University of Chicago over the past decade to identify hospitalized patients at risk for acute clinical deterioration. A simple (logistic regression based) ML model (eCARTv2) is commercially available within electronic health records on AgileMD’s clinical decision support platform. eCARTv2 was developed in a retrospective multicenter dataset and its use in clinical practice was associated with a 29% relative risk reduction in mortality in a multicenter trial. Our team recently completed development and validation of a gradient boosted machine (GBM) version of the model (eCARTv4), using nearly 100 variables, including trends and interactions. The advanced ML model was significantly more accurate than the simple ML and other models for predicting acute clinical deterioration across all hospital settings, in both septic and non-septic patients as well as in COVID-19 patients. The next challenge is clinically implementing it. The goals of this project are to a) upgrade the existing AgileMD platform to support the previously derived and validated eCARTv4 model and overhaul the human-machine interface for an advanced user experience (UX) that provides, for the first time, interpretable, graphical insight into the contribution of individual variables to a real-time EHR-embedded advanced ML analytic, and b) measure the impact of the new tool on HCP effectiveness, efficiency and satisfaction. We hypothesize that the combination of high accuracy and interpretability afforded by the advanced ML and UX will result in earlier recognition of acute deterioration as well as increased System Usability Scores (SUS) and usefulness scores in the treatment of deteriorating COVID-19 patients over standard care.

项目总结/摘要 先进的机器学习（ML）一直被证明优于专家意见，并且更简单。 用于预测临床结果的分析。然而，成功的前瞻性临床研究很少。 这些工具的使用。在医疗保健领域实施和采用高级ML的独特障碍 （1）在现有工作流程中实时运行和显示这些模型的技术挑战 以及（2）在高技能提供者中对黑盒算法的普遍不信任。因此， 这些工具在医疗保健中大量丢失。这在COVID-19中尤其成问题，患者可以 迅速恶化，从看似稳定到突然呼吸衰竭或休克，几乎没有明显的 警告及早认识到这种恶化对积极干预至关重要，这可以改善结果。 eCART是芝加哥大学过去反复开发的预测分析 十年来确定住院患者的急性临床恶化的风险。一个简单的（基于逻辑回归） ML模型（eCARTv 2）已在AgileMD临床决策的电子健康记录中上市 支持平台。eCARTv 2是在回顾性多中心数据集中开发的，并在临床实践中使用 在一项多中心试验中，与死亡率29%的相对风险降低相关。我们的团队最近完成了 开发和验证梯度增强机（GBM）版本的模型（eCARTv 4），使用近 100个变量，包括趋势和相互作用。先进的ML模型明显比 简单的ML和其他模型用于预测所有医院环境中的急性临床恶化， 脓毒症和非脓毒症患者以及COVID-19患者。下一个挑战是临床应用。 该项目的目标是：a）升级现有的AgileMD平台，以支持以前衍生的 经过验证的eCARTv 4模型，并彻底改造人机界面，以获得高级用户体验（UX） 这是第一次提供了可解释的图形化洞察力，以了解单个变量对 实时EHR嵌入式高级ML分析，以及B）测量新工具对HCP的影响 有效性、效率和满意度。我们假设，高准确性和 先进的ML和UX提供的可解释性也将导致对急性恶化的早期识别 在治疗不断恶化的COVID-19中，系统可用性评分（SUS）和有用性评分增加 患者的标准护理。