Reinforcement Learning for optimal treatment strategies in healthcare applications

强化学习在医疗保健应用中实现最佳治疗策略

• 批准号: 2440893
• 金额: --
• 依托单位: University of Warwick
• 依托单位国家: 英国
• 项目类别: Studentship
• 财政年份: 2020
• 资助国家: 英国
• 起止时间: 2020 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=studentship-2440893
• 关键词: Reinforcement; Learning; optimal; treatment; strategies

An optimal stopping problem aims at finding an optimal policy that describes the right time at which to take a particular action in a stochastic process, to maximize an expected reward.Here the focus is placed on applications in the healthcare sector and predictive models characterized by sequential decision-making settings.The aim is to investigate the use of RL algorithms to learn optimal stopping policies suitable to describe a sequential decision-making setting, typical in a medical intervention study. The problem of interest regards a prognostic model attempting to reduce the risk of an adverse outcome of a group of patients given a set of covariates. Similar use can be found determining when to stop the treatment of patients receiving fractionated radiotherapy treatments, but also in response-guided problems of pharmacological treatments.The stopping problem is introduced by the role of interventions (actions) taken by the medical practitioner (agent) that create a causal link affecting the predictions (environment), to reduce the patient's risk. In real-world settings it is also observed that interventions driven by the score can change the distribution of the data and outcomes, leading to a decay in observed performance, particularly if the intervention is successful. As a result, this requires learning an optimal policy to address the stopping problem and dealing with the causal process governing the 'intervened' covariate and the outcome.The challenges and novelties by the problem regard strategies to incorporate stochastic intervention functions by means of Gaussian Processes as well as constrained policy optimization to take into account of real-world approximation of constraints (e.g. resource allocation in a hospital facility) referred to as costs of opening, running, and closing the activities.Further developments and novelties can be sought in extending the investigation to a multi agent framework to model the collaborative/competitive relationship behaviour among, for example, managing of resources across hospitals.Ajdari, A., Niyazi, M., Nicolay, N.H., Thieke, C., Jeraj, R. and Bortfeld, T., 2019. Towards optimal stopping in radiation therapy. Radiotherapy and Oncology, 134, pp.96-100.Kotas, J., 2019. Optimal stopping for response-guided dosing. Networks & Heterogeneous Media, 14(1), p.43.Lenert, M.C., Matheny, M.E. and Walsh, C.G., 2019. Prognostic models will be victims of their own success, unless.... Journal of the American Medical Informatics Association, 26(12), pp.1645-1650.Deliu, N., Williams, J.J. and Chakraborty, B., 2022. Reinforcement Learning in Modern Biostatistics: Constructing Optimal Adaptive Interventions. arXiv preprint arXiv:2203.02605.Wu, S.A., Wang, R.E., Evans, J.A., Tenenbaum, J.B., Parkes, D.C. and Kleiman-Weiner, M., 2021. Too Many Cooks: Bayesian Inference for Coordinating Multi-Agent Collaboration. Topics in Cognitive Science, 13(2), pp.414-432.Titsias, M.K., Schwarz, J., Matthews, A.G.D.G., Pascanu, R. and Teh, Y.W., 2019. Functional regularisation for continual learning with gaussian processes. arXiv preprint arXiv:1901.11356.

最优停止问题的目标是找到一个最优策略，该策略描述了在随机过程中采取特定行动的正确时间，以最大化预期收益。这里的重点放在医疗保健部门的应用和以顺序决策设置为特征的预测模型上。目的是研究RL算法的使用，以学习适合于描述在医疗干预研究中典型的顺序决策设置的最优停止策略。感兴趣的问题涉及一个预后模型，该模型试图降低给定一组协变量的一组患者的不良结局的风险。类似的用法可以用来确定何时停止对接受分次放射治疗的患者的治疗，但也可以用于药物治疗的反应导向问题。停止问题是由医生(代理人)所采取的干预(行动)的作用引入的，这些干预(行动)产生了影响预测(环境)的因果联系，以降低患者的风险。在现实世界中，也可以观察到，由分数驱动的干预会改变数据和结果的分布，导致观察到的绩效下降，特别是在干预成功的情况下。因此，这需要学习一个最优策略来解决停止问题，并处理控制被干预的协变量和结果的因果过程。该问题的挑战和新颖性涉及通过高斯过程合并随机干预功能的策略以及考虑约束(例如医院设施中的资源分配)的现实世界近似的策略优化，所述约束被称为活动的开始、运行和结束的成本。在将调查扩展到多代理框架以对例如跨医院的资源管理之间的协作/竞争关系行为进行建模的过程中可以寻求进一步的发展和新颖性。A.，Niyazi，M.，Nicolay，N.H.，Thieke，C.，Jeraj R.和Bortfeld，T.，2019.在放射治疗中走向最佳停止。《放射治疗与肿瘤学》，134，第96-100页。响应引导剂量的最佳停止。网络与异质媒体，14(1)，第43页。Lenert，M.C.，Matheny，M.E.和Walsh，C.G.预测模型将成为自身成功的牺牲品，除非……《美国医学信息学协会杂志》，26(12)，1645-1650页。现代生物统计学中的强化学习：构建最优适应性干预措施。Arxiv预印本arxiv：2203.02605.Wu，S.A.，Wang，R.E.，Evans，J.A.，Tenenbaum，J.B.，Parkes，D.C.和Kleiman-Weiner，M.，2021。厨师太多：协调多智能体协作的贝叶斯推理。《认知科学专题》，13(2)，pp.414-432.Titsias，M.K.，Schwarz，J.，Matthews，A.G.D.G.，Pascanu，R.和Teh，Y.W.，2019。高斯过程连续学习的函数正则化。Arxiv预印本arxiv：1901.11356。