Reinforcement Learning for optimal treatment strategies in healthcare applications

最优停止问题的目标是找到一个最优策略，该策略描述了在随机过程中采取特定行动的正确时间，以最大化预期奖励。这里的重点放在医疗保健部门的应用程序和以顺序决策设置为特征的预测模型上。目的是研究RL算法的使用，以学习适合描述顺序决策设置的最佳停止策略，典型的在医疗干预研究中。我们感兴趣的问题是一个预后模型，它试图降低给定一组协变量的一组患者出现不良结果的风险。类似的用途可以发现，确定何时停止治疗的病人接受分割放疗治疗，但也在反应导向问题的药物治疗。停止问题是由医生（代理人）采取的干预措施（行动）的作用引起的，这些干预措施（行动）产生了影响预测（环境）的因果关系，以减少患者的风险。在现实环境中，我们还观察到，由分数驱动的干预措施可能会改变数据和结果的分布，导致观察到的性能下降，尤其是在干预措施成功的情况下。因此，这需要学习一种最优策略来解决停止问题，并处理控制“被干预”协变量和结果的因果过程。该问题的挑战和新颖之处涉及通过高斯过程结合随机干预函数的策略，以及考虑到现实世界的约束近似（例如，医院设施的资源分配）的约束优化，即开业、运行和关闭活动的成本。在将调查扩展到多主体框架以模拟协作/竞争关系行为（例如，跨医院资源管理）方面，可以寻求进一步的发展和新颖之处。Ajdari， A., Niyazi， M., Nicolay, n.h., Thieke， C., Jeraj， R.和Bortfeld， T., 2019。朝向最佳停止放射治疗的方向。放射与肿瘤，134,pp.96-100。J. Kotas, 2019。响应导向剂量的最佳停止。网络与异构媒体，14(1),p.43。Lenert, m.c., Matheny， M.E.和Walsh， c.g., 2019。预测模型将成为自身成功的受害者，除非....医学信息学报，26(12),pp.1645-1650。Deliu， N., Williams， J.J.和Chakraborty， B., 2022。现代生物统计学中的强化学习：构建最佳适应性干预。arXiv预印arXiv:2203.02605。Wu, s.a., Wang, r.e., Evans, j.a., Tenenbaum, j.b., Parkes， D.C.和Kleiman-Weiner， M., 2021。太多厨师：协调多智能体协作的贝叶斯推理。认知科学，13(2),pp.414-432。Titsias, m.k., Schwarz， J., Matthews, a.g.d.g., Pascanu， R.和Teh， y.w., 2019。高斯过程连续学习的函数正则化。arXiv预印本arXiv:1901.11356。

An optimal stopping problem aims at finding an optimal policy that describes the right time at which to take a particular action in a stochastic process, to maximize an expected reward.Here the focus is placed on applications in the healthcare sector and predictive models characterized by sequential decision-making settings.The aim is to investigate the use of RL algorithms to learn optimal stopping policies suitable to describe a sequential decision-making setting, typical in a medical intervention study. The problem of interest regards a prognostic model attempting to reduce the risk of an adverse outcome of a group of patients given a set of covariates. Similar use can be found determining when to stop the treatment of patients receiving fractionated radiotherapy treatments, but also in response-guided problems of pharmacological treatments.The stopping problem is introduced by the role of interventions (actions) taken by the medical practitioner (agent) that create a causal link affecting the predictions (environment), to reduce the patient's risk. In real-world settings it is also observed that interventions driven by the score can change the distribution of the data and outcomes, leading to a decay in observed performance, particularly if the intervention is successful. As a result, this requires learning an optimal policy to address the stopping problem and dealing with the causal process governing the 'intervened' covariate and the outcome.The challenges and novelties by the problem regard strategies to incorporate stochastic intervention functions by means of Gaussian Processes as well as constrained policy optimization to take into account of real-world approximation of constraints (e.g. resource allocation in a hospital facility) referred to as costs of opening, running, and closing the activities.Further developments and novelties can be sought in extending the investigation to a multi agent framework to model the collaborative/competitive relationship behaviour among, for example, managing of resources across hospitals.Ajdari, A., Niyazi, M., Nicolay, N.H., Thieke, C., Jeraj, R. and Bortfeld, T., 2019. Towards optimal stopping in radiation therapy. Radiotherapy and Oncology, 134, pp.96-100.Kotas, J., 2019. Optimal stopping for response-guided dosing. Networks & Heterogeneous Media, 14(1), p.43.Lenert, M.C., Matheny, M.E. and Walsh, C.G., 2019. Prognostic models will be victims of their own success, unless.... Journal of the American Medical Informatics Association, 26(12), pp.1645-1650.Deliu, N., Williams, J.J. and Chakraborty, B., 2022. Reinforcement Learning in Modern Biostatistics: Constructing Optimal Adaptive Interventions. arXiv preprint arXiv:2203.02605.Wu, S.A., Wang, R.E., Evans, J.A., Tenenbaum, J.B., Parkes, D.C. and Kleiman-Weiner, M., 2021. Too Many Cooks: Bayesian Inference for Coordinating Multi-Agent Collaboration. Topics in Cognitive Science, 13(2), pp.414-432.Titsias, M.K., Schwarz, J., Matthews, A.G.D.G., Pascanu, R. and Teh, Y.W., 2019. Functional regularisation for continual learning with gaussian processes. arXiv preprint arXiv:1901.11356.