Reinforcement Learning for optimal treatment strategies in healthcare applications

最优停止问题的目标是找到一个最优策略，该策略描述了在随机过程中采取特定行动的正确时间，最大化预期回报。这里的重点是放在医疗保健领域的应用和以顺序决策设置为特征的预测模型上。目的是研究使用RL算法来学习适合描述顺序决策的最优停止策略-这是一个典型的医疗干预研究。感兴趣的问题是关于一个预后模型，试图减少一组患者的一组协变量的不良后果的风险。类似的用途可以被发现确定何时停止接受分次放射治疗的患者的治疗，而且在药物治疗的响应引导问题中也是如此。停止问题是由医疗从业者（代理人）所采取的干预（行动）的作用引入的，这些干预（行动）创建了影响预测（环境）的因果关系，以降低患者的风险。在现实世界中，还观察到由分数驱动的干预可以改变数据和结果的分布，导致观察到的性能下降，特别是如果干预成功的话。因此，在本发明中，这需要学习一个最优策略来解决停止问题，并处理控制“干预”协变量和结果的因果过程。该问题的挑战和新颖性涉及通过高斯过程纳入随机干预函数的策略，以及考虑现实世界约束近似的约束策略优化（例如，医院设施中的资源分配），称为活动的开始、运行和结束的成本。可以在将调查扩展到多代理框架以模拟例如跨医院的资源管理之间的协作/竞争关系行为中寻求进一步的发展和新颖性。Niyazi，M.，Nicolay，N.H.，Thieke，C.，杰拉伊河和Bortfeld，T.，2019.放射治疗的最佳停止。放射治疗和肿瘤学，134，第96 - 100页。2019.响应引导给药的最佳停药。网络与异构媒体，14（1），第43页，Lenert，M.C.，Matheny，M.E.和沃尔什，C.G.，2019.预测模型将成为其自身成功的受害者，除非......美国医学信息学协会杂志，26（12），第1645 - 1650页。Deliu，N.，威廉姆斯，J. J.和Chakraborty，B.，2022.现代生物统计学中的强化学习：构建最佳适应性干预。arXiv预印本arXiv：2203.02605.Wu，S.A.，Wang，R.E.，埃文斯，J.A.，Tenenbaum，J.B.，帕克斯，哥伦比亚特区和克莱曼-韦纳，M.，2021. Too Many Cooks：Bayesian Inference for Coordinating Multi-Agent Collaboration（太多厨师：协调多代理协作的贝叶斯推理）《认知科学专题》，13（2），第414 - 432页。施瓦茨，J.，马修斯，总检察长，帕斯卡努河和Teh，Y.W.，2019.高斯过程连续学习的函数正则化。arXiv预印本arXiv：1901.11356。

An optimal stopping problem aims at finding an optimal policy that describes the right time at which to take a particular action in a stochastic process, to maximize an expected reward.Here the focus is placed on applications in the healthcare sector and predictive models characterized by sequential decision-making settings.The aim is to investigate the use of RL algorithms to learn optimal stopping policies suitable to describe a sequential decision-making setting, typical in a medical intervention study. The problem of interest regards a prognostic model attempting to reduce the risk of an adverse outcome of a group of patients given a set of covariates. Similar use can be found determining when to stop the treatment of patients receiving fractionated radiotherapy treatments, but also in response-guided problems of pharmacological treatments.The stopping problem is introduced by the role of interventions (actions) taken by the medical practitioner (agent) that create a causal link affecting the predictions (environment), to reduce the patient's risk. In real-world settings it is also observed that interventions driven by the score can change the distribution of the data and outcomes, leading to a decay in observed performance, particularly if the intervention is successful. As a result, this requires learning an optimal policy to address the stopping problem and dealing with the causal process governing the 'intervened' covariate and the outcome.The challenges and novelties by the problem regard strategies to incorporate stochastic intervention functions by means of Gaussian Processes as well as constrained policy optimization to take into account of real-world approximation of constraints (e.g. resource allocation in a hospital facility) referred to as costs of opening, running, and closing the activities.Further developments and novelties can be sought in extending the investigation to a multi agent framework to model the collaborative/competitive relationship behaviour among, for example, managing of resources across hospitals.Ajdari, A., Niyazi, M., Nicolay, N.H., Thieke, C., Jeraj, R. and Bortfeld, T., 2019. Towards optimal stopping in radiation therapy. Radiotherapy and Oncology, 134, pp.96-100.Kotas, J., 2019. Optimal stopping for response-guided dosing. Networks & Heterogeneous Media, 14(1), p.43.Lenert, M.C., Matheny, M.E. and Walsh, C.G., 2019. Prognostic models will be victims of their own success, unless.... Journal of the American Medical Informatics Association, 26(12), pp.1645-1650.Deliu, N., Williams, J.J. and Chakraborty, B., 2022. Reinforcement Learning in Modern Biostatistics: Constructing Optimal Adaptive Interventions. arXiv preprint arXiv:2203.02605.Wu, S.A., Wang, R.E., Evans, J.A., Tenenbaum, J.B., Parkes, D.C. and Kleiman-Weiner, M., 2021. Too Many Cooks: Bayesian Inference for Coordinating Multi-Agent Collaboration. Topics in Cognitive Science, 13(2), pp.414-432.Titsias, M.K., Schwarz, J., Matthews, A.G.D.G., Pascanu, R. and Teh, Y.W., 2019. Functional regularisation for continual learning with gaussian processes. arXiv preprint arXiv:1901.11356.