CAREER: Real-time Control of Cell Differentiation Using Reinforcement Learning

正在开发治疗性细胞来治疗慢性疾病。这些细胞通常来源于干细胞。在这项技术可以在临床试验之外使用之前，必须做两件事。首先，必须提高治疗细胞的质量。其次，必须改进这些细胞的制造，使所有细胞都具有必要的治疗活性水平。大规模生产治疗细胞的复制是该项目的最终目标。该项目测试了主动控制将提高治疗性细胞制造的再现性的假设。这项研究还将通过互动艺术展览提高学生和公众对机器学习（ML）方法的理解。这项工作将使再生药物的实际生产以及基于ML的数值方法和工程师过程控制策略的教育成为可能。该项目将培养研究生和10至20名本科生，掌握工具设计、细胞培养和强化学习所需的技能，从而加强我国的生物制造能力。该项目将开发一个模块化框架，该框架可适用于任何途径，并可随着新传感器的开发、刺激线索的发现和新细胞靶点的确定而更新。三个研究目标将确定动态控制和RL训练迭代次数对区分改进的影响。这些是（1）构建控制框架，（2）启用控制框架的训练，以及（3）针对模型单元上的静态配方来基准动态控制。四个模型细胞系跨越三个胚层目标，并建立了化学，物理和电分化配方。在第一个目标中，强化学习代理将构建在TensorFlow包上。马尔可夫（无记忆）和非马尔可夫奖励函数将与最新的RL算法一起在基于文献的假设构建的计算机模拟器上进行沿着测试。 与此同时，第二个目标侧重于开发一个真实的训练环境，该环境包括沿着三个控制元素（化学、物理和电）的多种传感模态。这些将是可并行化的环境，以允许RL代理的有效训练。在目标3中，RL训练的动态控制策略将针对静态配方进行批内和批间变异性基准测试。 这些实验将确定这种动态控制是否能提高制造一致性，并同样提供训练新的差异化路线所需的平均时间。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Therapeutic cells are being developed to treat chronic diseases. These cells are most often derived from stem cells. Two things must happen before this technology can be made available outside of clinical trials. First, the quality of the therapeutic cells must be improved. Second, the manufacture of these cells must be improved so all cells have a necessary level of therapeutic activity. Reproducibility in large scale manufacturing of therapeutic cells is the ultimate objective of this project. This project tests the hypothesis that active control will improve the reproducibility of therapeutic cell manufacture. This research will also improve student and public understanding of machine learning (ML) approaches through an interactive art exhibit. This work will enable practical manufacture of regenerative medicines as well as education of ML-based numerical methods and process control strategies for engineers. This project will train graduate students and 10 to 20 undergraduates with the skills required for tool design, cell culture, and reinforcement learning, thereby strengthening our nation’s biomanufacturing capabilities.This project will develop a modular framework that can be applied to any pathway, and can be updated as new sensors are developed, stimulation cues are discovered, and new cell targets are identified. Three research aims will determine the impacts of dynamic control and RL training iteration number on differentiation improvement. These are (1) build the control framework, (2) enable training of the control framework, and (3) benchmark dynamic control against static recipes on model cells. Four model cell lines span three germ layer targets and have established chemical, physical, and electrical differentiation recipes. In the first aim, the reinforcement learning agent will be built on the TensorFlow package. Markov (memoryless) and non-Markov reward functions will be tested along with latest RL algorithms on an in silico simulator built from literature-based assumptions. In parallel, the second aim focuses on development of a real training environment that includes multiple sensing modalities along with three control elements (chemical, physical, and electrical). These will be parallelizable environments to allow for efficient training of the RL agent. In Aim 3, the RL-trained dynamic control strategy will be benchmarked against static recipes for intra- and inter- batch variability. These experiments will determine if such dynamic control improves manufacture consistency and will likewise provide average time needed for training a new differentiation route.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.