Addressing Algorithmic Unreliability and Dataset Shift in EHR-based Risk Prediction Models

项目摘要 基于电子健康记录(EHR)输入的预测分析算法，例如患者特征、 管理代码和实验室值越来越多地被用于医疗保健环境中，以将资源引导到高- 有风险的病人。数据在开发和部署有效的预测模型方面发挥着不可或缺的作用。 维护这些工具面临的最大挑战是与数据有关的问题， 即数据集转移，其中训练数据分布不同于算法所在的种群 部署，导致模型恶化和不准确的风险预测。数据集移动是一种普遍的原因 由于医生行为和健康的不可避免变化，基于EHR的模型中的算法不可靠 改变(1)输入分布(协变量漂移)的系统操作；以及(2)改变 预测因素和结果(概念漂移)。新冠肺炎大流行期间医疗服务利用的突然变化 可能影响了数据生成过程和临床预测模型的性能。我们的 初步研究表明，新冠肺炎隔离期间患者实验室收集的减少导致 基于EHR的单机构死亡风险预测的重要预测因子稀疏数据生成 算法，低估了晚期癌症患者的风险。尽管预测性工具的使用越来越多 在高风险的临床应用中；以及对数据集转移的日益认识，我们缺乏推理的框架 班次及其对护理提供的影响；以及主动应对班次以保持随时间推移的绩效。在……里面 目标1，我们建议将之前的工作扩展到全国部署的风险预测算法--VA CARE 评估需要(CAN)模型，每年用于数百万退伍军人管理局受益人。VA CAN模型预测 初级保健后90天内或1年内住院以确定高危人群的可能性 将从额外的门诊干预中受益的患者。我们还研究了协变量和概念 漂移是新冠肺炎关联数据集漂移的两种可能机制。在目标2中，我们应用中断时间 研究大流行初期突变与病例管理之间关系的系列设计 决定。目前解决数据集转移的解决方案主要是被动解决方案(即，通过 最近的数据)，然而，在新的测试环境中不是很健壮。在目标3中，我们考虑修订退伍军人事务部 可以通过机器学习和包含反映潜在换班驱动因素的变量进行建模。这个项目是 创新，因为它是第一个利用严格的统计框架来研究转移程度和机制的公司 并为模型维护制定积极的指导方针。对于医学博士科拉来说，培训计划是严格的 生物统计学专业的学生。她得到了她所在部门和机构的大力支持，以及她的两个高中生- 合格赞助商：EHR风险预测建模专家陈劲波博士和 预测分析实施方面的专家。拟议的研究和职业发展计划将是 这是科拉女士成为一名跨学科和独立的内科科学家的关键一步。

Project Summary Predictive analytic algorithms built on electronic health record (EHR) inputs, such as patient characteristics, administrative codes, and lab values, are increasingly used in health care settings to direct resources to high- risk patients. Data play an indispensable role in the development and deployment of effective predictive models. The greatest, yet understudied, challenge in the maintenance of these tools arises from a data-related concern, namely dataset shift, in which training data distribution differs from the population on which the algorithm is deployed, leading to model deterioration and inaccurate risk predictions. Dataset shift is a pervasive cause of algorithmic unreliability in EHR-based models due to inevitable changes in physician behaviors and health system operations that alter (1) the input distribution (covariate drift); and (2) changes in the relationship between predictors and outcome (concept drift). Sudden changes in healthcare utilization during the COVID-19 pandemic may have impacted the data generation process and the performance of clinical predictive models. Our preliminary study showed that decreased collection of patient labs during the COVID-19 quarantine period led to sparse data generation for important predictors of a single-institution EHR-based mortality risk prediction algorithm, underpredicting risk for patients with advanced cancers. Despite the increasing use of predictive tools in high stakes clinical applications; and growing recognition of dataset shift, we lack a framework for reasoning shift and its effects on care delivery; and for proactively addressing shift to maintain performance over time. In Aim 1, we propose to extend prior works on shift to a nationally deployed risk prediction algorithm, the VA Care Assessment Need (CAN) model, used on millions of VA beneficiaries each year. The VA CAN model predicts the likelihood of hospitalization within 90 days or 1 year after a primary care encounter to identify high-risk patients who would benefit from additional outpatient interventions. We also investigate covariate and concept drift as two possible mechanisms for COVID-19 associated dataset shift. In Aim 2, we apply an interrupted time series design to study the association between sudden shift at the onset of the pandemic on case-management decisions. Current solutions to address dataset shift have primarily been reactive (i.e. model retraining with recent data), however, fail to be robust in new testing environments. In Aim 3, we consider revision of the VA CAN model via machine learning and inclusion of variables that reflect potential drivers of shift. This project is innovative as it is the first to leverage a rigorous statistical framework to study extent and mechanisms of shift and develop proactive guidelines for model maintenance. The training plan is rigorous for Ms. Kolla, an MD-PhD student in biostatistics. She is strongly supported by her department and institution as well as her two high- qualified sponsors: Dr. Jinbo Chen, an expert in EHR-based risk prediction modeling, and Dr. Ravi Parikh, an expert in implementation of predictive analytics. The proposed research and career development plan will be an essential step towards Ms. Kolla’s development as an interdisciplinary and independent physician-scientist.