Multi-Modal Reinforcement Learning Algorithms for Improving Context-Sensitive Closed-Loop Blood Glucose Control for Type 1 Diabetics

用于改善 1 型糖尿病患者上下文敏感闭环血糖控制的多模态强化学习算法

• 批准号: 2452234
• 金额: --
• 依托单位: University of Bristol
• 依托单位国家: 英国
• 项目类别: Studentship
• 财政年份: 2020
• 资助国家: 英国
• 起止时间: 2020 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=studentship-2452234
• 关键词: Multi; Modal; Reinforcement; Learning; Algorithms

Historically, type 1 diabetics must adhere to a strict regime of blood glucose monitoring and daily self-administered insulin injections to maintain their glucose levels within a healthy range. The advent of continuous glucose monitors (CGM) and insulin pumps has significantly improved the capabilities of type 1 diabetics in managing their condition. However, the majority of the burden for interpreting the data and selecting the insulin dosage still falls on the individual. Hybrid closed-loop insulin delivery systems are already available to meet this need, using control algorithms to automatically adjust basal insulin infusion rates based on blood glucose measurements. These systems have shown success in improving glucose levels and reducing glycaemic variability (Lal et al. 2019, McAuley et al. 2020), but are limited in their ability to handle instances in which blood glucose changes rapidly. To address this, reinforcement learning (RL) algorithms have been utilised to retroactively respond to changes in blood glucose. These algorithms consider states consisting of glucose, insulin, and carbohydrate data and apply this information to select the optimum basal or bolus insulin dosage. RL algorithms have been tested in silico, yielding improvements in mean glucose levels when compared to commercial control algorithms (Yamagata et al. 2020, Fox et al. 2020). However, these approaches have relied on virtual patient cohorts for algorithmic training, which are limited in their application to real clinical populations. Furthermore, type 1 diabetics utilise a wealth of information beyond the described state spaces of current RL approaches, using knowledge relating to activity, stress, and illness to further inform their decision making. This project will build on state-of-the-art RL algorithms for glucose control and incorporate novel indicators of blood glucose from real patient data, in order to improve the suitability of these algorithms for clinical application. This approach will develop model based RL (MBRL) algorithms for offline learning and then train them on available sources of CGM and insulin data. The preliminary stages of the project will consist of analysing these sources and assessing their limitations. This will focus on datasets containing passively collected variables which could be indicative of periods of stress, work, activity, non-engagement, or illness, such as those present in the OhioT1DM (Marling and Bunesca, 2020) or D1NAMO datasets (Dubosson et al. 2020). If the available data is insufficient, this will be concluded by a period of data collection consisting of adult participants each using a CGM and insulin pump. The participants' blood glucose levels, insulin usage and food intake will be continuously recorded over a 4-week period, with additional measurements of factors such as heart rate and step count being logged using commercial wearables. To utilise the small CGM datasets effectively, techniques for improving algorithmic sample-efficiency will also be explored. This will include augmenting the datasets using methods such as contrastive learning or data generation, modifying existing sample efficient RL algorithms, such as shallow MBRL or Bayesian RL and utilising transfer learning to train models on virtual patient cohorts and applying them to real patient data.The performance of the RL algorithm will be evaluated in silico using trajectory inspection (Ji et al, 2020), by comparing the projected blood glucose to those achieved by the patient in the dataset. Following successful implementation, the algorithm will be adapted to improve its clinical practicality. This could include exploring methods for reducing the risk and burden associated with the algorithmic training process, introducing options for customised control based on user suggestions or modifying the algorithm to provide varying levels of control when applied to patients who are not engaged with the management of their diabetes.

从历史上看，1型糖尿病患者必须坚持严格的血糖监测制度和每天自我注射胰岛素，以将他们的血糖水平保持在健康的范围内。连续血糖监测仪(CGM)和胰岛素泵的出现极大地提高了1型糖尿病患者管理病情的能力。然而，解释数据和选择胰岛素剂量的大部分负担仍然落在个人身上。混合式闭环式胰岛素输注系统已经可以满足这一需求，使用控制算法根据血糖测量自动调整基础胰岛素输注速率。这些系统在改善血糖水平和降低血糖变异性方面取得了成功(Lal等人。2019年，McAuley等人。2020)，但处理血糖快速变化的情况的能力有限。为了解决这个问题，强化学习(RL)算法已经被用于追溯响应血糖的变化。这些算法考虑由葡萄糖、胰岛素和碳水化合物数据组成的状态，并应用这些信息来选择最佳的基础或团注胰岛素剂量。RL算法已经在Silico中进行了测试，与商业控制算法相比，平均血糖水平得到了改善(Yamagata等人。2020年，福克斯等人。2020)。然而，这些方法依赖于虚拟患者队列进行算法训练，这限制了它们在真实临床人群中的应用。此外，1型糖尿病患者利用当前RL方法所描述的状态空间之外的丰富信息，使用与活动、压力和疾病相关的知识来进一步指导他们的决策。该项目将建立在最先进的血糖控制RL算法的基础上，并结合来自真实患者数据的新的血糖指标，以提高这些算法在临床应用的适用性。这种方法将开发用于离线学习的基于模型的RL(MBRL)算法，然后根据可用的CGM和胰岛素数据来源对它们进行训练。该项目的初步阶段将包括分析这些来源并评估其局限性。这将侧重于包含被动收集的变量的数据集，这些变量可能指示压力、工作、活动、不参与或疾病的时期，例如俄亥俄州T1 DM(Marling和Bunesca，2020)或D1NAMO数据集(DuBosson等人)中存在的那些变量。2020)。如果现有数据不充分，将通过一段时间的数据收集结束，该阶段由成年参与者组成，每人使用CGM和胰岛素泵。参与者的血糖水平、胰岛素使用量和食物摄入量将在4周内持续记录，并使用商业可穿戴设备记录心率和步数等因素的额外测量。为了有效地利用小的CGM数据集，还将探索提高算法样本效率的技术。这将包括使用对比学习或数据生成等方法来扩充数据集，修改现有的样本高效的RL算法，例如浅MBRL或贝叶斯RL，并利用转移学习来训练虚拟患者队列的模型并将其应用于真实患者数据。RL算法的性能将在使用轨迹检测的计算机中进行评估(纪万昌等人，2020)，方法是将预测的血糖与患者在数据集中实现的血糖进行比较。在成功实施后，将对该算法进行调整，以提高其临床实用性。这可能包括探索减少与算法培训过程相关的风险和负担的方法，根据用户建议引入定制控制的选项，或者修改算法，以便在应用于不参与糖尿病管理的患者时提供不同程度的控制。