Engaging Multidisciplinary Health System Stakeholders to Create a Process for Implementing Machine-Learning Enabled Clinical Decision Support

项目概要/摘要 用于临床决策支持 (CDS) 的“黑匣子”机器学习 (ML) 模型的激增引起了人们的关注。 关于 CDS 可解释性、可操作性和整体可用性的担忧，迫切需要一个明确的 让包括开发人员和用户在内的各种利益相关者参与实施规划的过程。 我们的长期目标是制定一个流程来指导卫生系统进行规划、监测和评估 CDS 实施。 R21 的总体目标是开发和评估通用策略 在 CDS 探索阶段将多学科利益相关者聚集在一起，以确定 在其背景下实施的促进因素和障碍。为此，我们将使用参与式系统 动力学（PSD）建模作为一种多组件策略来评估和规划实施 利益相关者在实施的探索阶段，当决策发生时，以某种方式 支持 ML 的 CDS 可以随着时间的推移而持续。因此，我们将重点关注上游实施 支持 ML 的 CDS 的可接受性、适当性和可行性的结果。该项目的理由是 尽早让不同利益相关者参与实施规划的流程将澄清承诺 通过揭示可接受性、可行性和适当性来评估实施和采用的潜力。对于 在这个项目中，我们将重点关注一组特定的支持 ML 的 CDS：早期预警评分 (EWS)，用于 识别失代偿患者。我们计划通过实现两个具体目标来实现我们的总体目标：1. 让多学科利益相关者参与 EWS 实施（用户、开发人员、实施者、 所有者）从两个系统不同的采用环境中共同定义关键的共同障碍和促进因素 使用组模型构建 CDS 的可接受性、适当性和可行性的实施结果 系统动力学领域的脚本和 2. 通过测量变化来评估 PSD 过程 承诺采用 CDS（使用可接受性、适当性和可行性的衡量标准），引发 反馈并估计干预力度。我们将通过一系列小组建模会议获取数据 在不同环境中使用 CDS 的利益相关者，其中警报因目标用户、时间和频率而异 除其他因素外。我们将使用系统动力学建模领域定义明确的脚本来 促进小组讨论，以制定关于支持 ML 的 CDS 响应问题的共享理论 （目标 1）。由于任何策略的实施都需要适应，因此我们将评估 PSD 流程（Aim 2）完善并准备在其他地方使用。这一贡献意义重大，因为 EWS 得到了广泛使用 涵盖学术医院和社区医院。这项贡献通过使用群体建模而具有创新性 解决基于机器学习的 CDS 实施问题的技术，创造了方法论和实质性 调查结果。未来的 R01 将前瞻性地评估在多个用例设置中使用此流程的好处 同时继续构建供下游采用的因素的动态系统模型。

PROJECT SUMMARY/ABSTRACT The proliferation of “black box” Machine Learning (ML) models for Clinical Decision Support (CDS) has raised concerns regarding CDS interpretability, actionability and overall usability, rendering a critical need for a clear process that engages various stakeholders including both developers and users in implementation planning. Our long-term goal is to formalize a process to guide health systems in planning, monitoring and evaluating CDS implementation. The overall objective for this R21 is to develop and evaluate a generalizable strategy to bring multidisciplinary stakeholders together during the CDS exploration phase to identify facilitators and barriers to implementation in their contexts. In doing so, we will use Participatory System Dynamics (PSD) modeling as a multi-component strategy to evaluate and plan implementation with stakeholders during the exploration phase of implementation, when decision-making occurs, in a way where ML-enabled CDS can be sustained over time. As such, we will focus on the upstream implementation outcomes of acceptability, appropriateness, and feasibility of ML-enabled CDS. The rationale for this project is that a process that engages diverse stakeholders in implementation planning early on will clarify commitment to implementation and potential for adoption by revealing acceptability, feasibility, and appropriateness. For this project we will focus on one particular set of ML-enabled CDS: Early Warning Scores (EWSs), used to identify decompensating patients. We plan to accomplish our overall objective by pursuing two specific aims: 1. Engage multidisciplinary stakeholders involved in EWS implementation (users, developers, implementers, owners) from two systematically varying adoption contexts to co-define common barriers and facilitators to key implementation outcomes of CDS acceptability, appropriateness, and feasibility using group model building scripts from the field of system dynamics and 2. Evaluate the PSD process by measuring change in commitment to adopt CDS (using measures of acceptability, appropriateness, and feasibility), eliciting feedback, and estimating intervention effort. We will obtain data via a series of group modeling sessions from stakeholders who have used CDS in different contexts, where alerts vary by target user, time, and frequency among other factors. We will employ well-defined scripts from the field of System Dynamics modeling to facilitate group discussion toward developing a shared theory about the problem of ML-enabled CDS response (Aim 1). Because implementation of any strategy requires adaptation, we will evaluate the PSD process (Aim 2) to refine and prepare for use elsewhere. This contribution is significant because EWSs are widely used across both academic and community hospitals. This contribution is innovative by using group modeling techniques for the problem of ML-enabled CDS implementation, creating both methodological and substantive findings. A future R01 will prospectively assess benefits of using this process in multiple use case settings while continuing to build out the dynamic systems model of factors for downstream adoption.