Multi-Modal Reinforcement Learning Algorithms for Improving Context-Sensitive Closed-Loop Blood Glucose Control for Type 1 Diabetics

点击翻译按钮获取中文摘要

Historically, type 1 diabetics must adhere to a strict regime of blood glucose monitoring and daily self-administered insulin injections to maintain their glucose levels within a healthy range. The advent of continuous glucose monitors (CGM) and insulin pumps has significantly improved the capabilities of type 1 diabetics in managing their condition. However, the majority of the burden for interpreting the data and selecting the insulin dosage still falls on the individual. Hybrid closed-loop insulin delivery systems are already available to meet this need, using control algorithms to automatically adjust basal insulin infusion rates based on blood glucose measurements. These systems have shown success in improving glucose levels and reducing glycaemic variability (Lal et al. 2019, McAuley et al. 2020), but are limited in their ability to handle instances in which blood glucose changes rapidly. To address this, reinforcement learning (RL) algorithms have been utilised to retroactively respond to changes in blood glucose. These algorithms consider states consisting of glucose, insulin, and carbohydrate data and apply this information to select the optimum basal or bolus insulin dosage. RL algorithms have been tested in silico, yielding improvements in mean glucose levels when compared to commercial control algorithms (Yamagata et al. 2020, Fox et al. 2020). However, these approaches have relied on virtual patient cohorts for algorithmic training, which are limited in their application to real clinical populations. Furthermore, type 1 diabetics utilise a wealth of information beyond the described state spaces of current RL approaches, using knowledge relating to activity, stress, and illness to further inform their decision making. This project will build on state-of-the-art RL algorithms for glucose control and incorporate novel indicators of blood glucose from real patient data, in order to improve the suitability of these algorithms for clinical application. This approach will develop model based RL (MBRL) algorithms for offline learning and then train them on available sources of CGM and insulin data. The preliminary stages of the project will consist of analysing these sources and assessing their limitations. This will focus on datasets containing passively collected variables which could be indicative of periods of stress, work, activity, non-engagement, or illness, such as those present in the OhioT1DM (Marling and Bunesca, 2020) or D1NAMO datasets (Dubosson et al. 2020). If the available data is insufficient, this will be concluded by a period of data collection consisting of adult participants each using a CGM and insulin pump. The participants' blood glucose levels, insulin usage and food intake will be continuously recorded over a 4-week period, with additional measurements of factors such as heart rate and step count being logged using commercial wearables. To utilise the small CGM datasets effectively, techniques for improving algorithmic sample-efficiency will also be explored. This will include augmenting the datasets using methods such as contrastive learning or data generation, modifying existing sample efficient RL algorithms, such as shallow MBRL or Bayesian RL and utilising transfer learning to train models on virtual patient cohorts and applying them to real patient data.The performance of the RL algorithm will be evaluated in silico using trajectory inspection (Ji et al, 2020), by comparing the projected blood glucose to those achieved by the patient in the dataset. Following successful implementation, the algorithm will be adapted to improve its clinical practicality. This could include exploring methods for reducing the risk and burden associated with the algorithmic training process, introducing options for customised control based on user suggestions or modifying the algorithm to provide varying levels of control when applied to patients who are not engaged with the management of their diabetes.