Reinforcement Learning on the Edge: Specialised Control Policies with Limited Resources

近年来，人们对强化学习（RL）的现实应用越来越感兴趣。一个突出的领域是移动的机器人。机器人包括传感器和执行器，分别在环境中观察和行动。即使这些设备根据通过RL学习的策略进行控制，它们通常也必须依赖于集中式服务器来确定其下一步操作。这大大增加了机器人的延迟，使RL成为现实世界部署的不可行解决方案。此外，机器人可能会遇到与它们被训练的情况不同的情况。本项目旨在解决这两个问题。首先，它旨在为RL模型实现有效的设备上推理。其次，它将研究终身RL的新方法，可以动态地从新的经验中学习。通过这些线的调查，这个项目旨在引入一个新的RL范式，可以利用多个同时资源受限的学习者的经验和处理能力。这些学习者的任务是评估RL政策，并以资源有效的方式将新的观察结果整合到他们的模型中。学习者应该能够定期与其他设备共享这些信息，类似于现有的分布式RL方法。新的学习方法和RL算法将需要开发这个目标的实现。它将系统和机器学习的工作结合在一个新的环境中。

In recent years, there has been increasing interest in real-world applications of Reinforcement Learning (RL). One prominent domain is that of mobile robots. Robots include sensors and actuators to observe and act in environments, respectively. Even when such devices are controlled according to a policy learned through RL, they typically must rely on a centralised server to determine their next actions. This significantly increases the robot's latency and makes RL an infeasible solution for real-world deployment. Furthermore, robots may encounter situations which differ from those on which they were trained. This project aims to solve both of these problems. Firstly, it seeks to implement efficient on-device inference for RL models. Secondly, it will investigate new methods of lifelong RL which can dynamically learn from new experiences. Through these lines of investigation, this project seeks to introduce a new paradigm of RL which can leverage the experiences and processing power of multiple simultaneous resource-constrained learners. These learners will be tasked both with evaluating RL policies and with integrating new observations into their models in a resource-efficient way. Learners should be able to share this information with other devices periodically, similar to existing distributed RL approaches. New learning approaches and RL algorithms will need to be developed for this goalto be realised. It combines work from systems and machine learning in a novel context.