Multi-Modal Reinforcement Learning Algorithms for Improving Context-Sensitive Closed-Loop Blood Glucose Control for Type 1 Diabetics

从历史上看，1型糖尿病患者必须坚持严格的血糖监测制度，每天自我注射胰岛素，以将血糖水平维持在健康范围内。连续血糖监测仪（CGM）和胰岛素泵的出现显著提高了1型糖尿病患者管理病情的能力。然而，解释数据和选择胰岛素剂量的大部分负担仍然落在个人身上。混合型闭环胰岛素输送系统已经可以满足这一需求，使用控制算法根据血糖测量值自动调整基础胰岛素输注速率。这些系统在改善血糖水平和降低血糖变异性方面取得了成功（Lal et al. 2019, McAuley et al. 2020），但在处理血糖快速变化的情况下，它们的能力有限。为了解决这个问题，强化学习（RL）算法被用来对血糖的变化做出追溯反应。这些算法考虑由葡萄糖、胰岛素和碳水化合物数据组成的状态，并应用这些信息来选择最佳的基础或大剂量胰岛素。强化学习算法已经在计算机上进行了测试，与商业控制算法相比，平均血糖水平有所改善（Yamagata et al. 2020, Fox et al. 2020）。然而，这些方法依赖于虚拟患者队列进行算法训练，这在实际临床人群中的应用受到限制。此外，1型糖尿病患者利用当前RL方法所描述的状态空间之外的大量信息，利用与活动、压力和疾病相关的知识进一步为他们的决策提供信息。该项目将以最先进的RL血糖控制算法为基础，并结合来自真实患者数据的新型血糖指标，以提高这些算法在临床应用中的适用性。该方法将开发用于离线学习的基于模型的强化学习（MBRL）算法，然后在可用的CGM和胰岛素数据来源上训练它们。该项目的初步阶段将包括分析这些来源和评估其局限性。这将侧重于包含被动收集变量的数据集，这些变量可以指示压力、工作、活动、不参与或疾病的时期，例如存在于OhioT1DM （Marling和Bunesca， 2020）或D1NAMO数据集（Dubosson等人，2020）中的变量。如果现有数据不足，将通过一段时间的数据收集来得出结论，该数据收集由使用CGM和胰岛素泵的成年参与者组成。参与者的血糖水平、胰岛素使用情况和食物摄入量将在四周内持续记录下来，并使用商用可穿戴设备记录心率和步数等额外因素。为了有效地利用小型CGM数据集，还将探索提高算法样本效率的技术。这将包括使用对比学习或数据生成等方法来增加数据集，修改现有的样本高效强化学习算法，如浅MBRL或贝叶斯强化学习，利用迁移学习在虚拟患者队列上训练模型，并将其应用于真实患者数据。RL算法的性能将通过轨迹检查（Ji等人，2020）在计算机上进行评估，方法是将预测的血糖与数据集中患者的血糖进行比较。在成功实施后，将对该算法进行调整，以提高其临床实用性。这可能包括探索减少与算法培训过程相关的风险和负担的方法，根据用户建议引入定制控制选项，或修改算法，以便在应用于不参与糖尿病管理的患者时提供不同程度的控制。

Historically, type 1 diabetics must adhere to a strict regime of blood glucose monitoring and daily self-administered insulin injections to maintain their glucose levels within a healthy range. The advent of continuous glucose monitors (CGM) and insulin pumps has significantly improved the capabilities of type 1 diabetics in managing their condition. However, the majority of the burden for interpreting the data and selecting the insulin dosage still falls on the individual. Hybrid closed-loop insulin delivery systems are already available to meet this need, using control algorithms to automatically adjust basal insulin infusion rates based on blood glucose measurements. These systems have shown success in improving glucose levels and reducing glycaemic variability (Lal et al. 2019, McAuley et al. 2020), but are limited in their ability to handle instances in which blood glucose changes rapidly. To address this, reinforcement learning (RL) algorithms have been utilised to retroactively respond to changes in blood glucose. These algorithms consider states consisting of glucose, insulin, and carbohydrate data and apply this information to select the optimum basal or bolus insulin dosage. RL algorithms have been tested in silico, yielding improvements in mean glucose levels when compared to commercial control algorithms (Yamagata et al. 2020, Fox et al. 2020). However, these approaches have relied on virtual patient cohorts for algorithmic training, which are limited in their application to real clinical populations. Furthermore, type 1 diabetics utilise a wealth of information beyond the described state spaces of current RL approaches, using knowledge relating to activity, stress, and illness to further inform their decision making. This project will build on state-of-the-art RL algorithms for glucose control and incorporate novel indicators of blood glucose from real patient data, in order to improve the suitability of these algorithms for clinical application. This approach will develop model based RL (MBRL) algorithms for offline learning and then train them on available sources of CGM and insulin data. The preliminary stages of the project will consist of analysing these sources and assessing their limitations. This will focus on datasets containing passively collected variables which could be indicative of periods of stress, work, activity, non-engagement, or illness, such as those present in the OhioT1DM (Marling and Bunesca, 2020) or D1NAMO datasets (Dubosson et al. 2020). If the available data is insufficient, this will be concluded by a period of data collection consisting of adult participants each using a CGM and insulin pump. The participants' blood glucose levels, insulin usage and food intake will be continuously recorded over a 4-week period, with additional measurements of factors such as heart rate and step count being logged using commercial wearables. To utilise the small CGM datasets effectively, techniques for improving algorithmic sample-efficiency will also be explored. This will include augmenting the datasets using methods such as contrastive learning or data generation, modifying existing sample efficient RL algorithms, such as shallow MBRL or Bayesian RL and utilising transfer learning to train models on virtual patient cohorts and applying them to real patient data.The performance of the RL algorithm will be evaluated in silico using trajectory inspection (Ji et al, 2020), by comparing the projected blood glucose to those achieved by the patient in the dataset. Following successful implementation, the algorithm will be adapted to improve its clinical practicality. This could include exploring methods for reducing the risk and burden associated with the algorithmic training process, introducing options for customised control based on user suggestions or modifying the algorithm to provide varying levels of control when applied to patients who are not engaged with the management of their diabetes.