Towards Open-ended Reinforcement Learning using Synthetic Environment Generation

序贯决策问题在工程和科学中无处不在。强化学习（RL）是一种人工智能范式，其中代理人通过与环境的试错交互来学习决策技能，在处理复杂的决策任务方面取得了重大成功。然而，这些代理通常难以概括，在以前看不见的任务上表现出次优性能。此外，一旦部署，在新任务中表现不佳的人工智能往往缺乏改进的机会，因为它的学习过程在初始训练阶段之后就停止了。这些挑战限制了此类系统在现实世界中的实际应用，例如sim 2 real。开放式学习（OEL）旨在克服这些限制，其目标是产生对设计和训练期间未明确考虑的情况具有鲁棒性的学习系统。该提案特别侧重于通过合成环境生成的新技术开发自动化课程方法。因此，这个建议的目标分解如下：开发新的方法生成合成任务/environments.Use合成环境生成开发新的无监督环境设计（UED）方法forautomatic环境courses generation.Show，应用这样的课程，代理培训产生代理与强大的分布generalization.新奇的研究方法建议的研究主要着眼于扩展工作的无监督环境设计（UED）的发展子领域，旨在生成环境适合当前的学习代理，以促进continuedlearning。然而，UED目前仅限于生成级别，这些级别是特定任务的配置，在导航任务中迷宫的布局。我们寻求改进最先进的UED方法来生成整个环境，而不仅仅是水平，这是一个更一般的代理训练和解决的新任务。这样做将需要更复杂的生成AI技术，我们希望利用最新的进展，如扩散方法。最终，这项工作将在多个人工智能子领域的交叉点上进行操作，在高水平上即RLand生成AI。与EPSRC的战略和研究领域保持一致拟议的工作福尔斯属于EPSRC的人工智能技术职权范围。它在许多方面与EPSRC的目标保持一致，包括开发可在真实的世界环境中部署的新AI技术。我们还将在多个AI子领域的交叉点上工作，这与EPSRC支持跨学科研究方法的目标一致。

Sequential decision making problems are ubiquitous in engineering and science. Reinforcement Learning (RL), anAI paradigm where agents learn decision-making skills via trial-and-error interactions with their environment, hasachieved significant success in handling complex decision tasks. However, these agents often struggle to generalize,exhibiting suboptimal performance on previously unseen tasks. Furthermore, once deployed, an AI that underperforms in a new task often lacks opportunities for improvement, as its learning process ceases after the initialtraining phase. These challenges restrict the practical use of such systems in real-worlds settings, such as sim2real.Open-ended Learning (OEL) seeks to overcome these limitations with the goal of producing learning systems thatare robust to situations not explicitly considered during design and training.Aims and ObjectivesWhile there are many possible directions towards achieving OEL agents, this proposal specifically focuses on thedevelopment of automated curricula methods through novel techniques for synthetic environment generation. Assuch, the goals of this proposal are decomposed as follows:Develop new methods for generating synthetic tasks/environments.Use synthetic environment generation to develop novel Unsupervised Environment Design (UED) methods forautomatic environment curricula generation.Show that applying such curricula to agent training produces agents with strong out-of-distribution generalization.Novelty of the Research MethodologyThe proposed research primarily looks to extend work in the developing subfield of Unsupervised EnvironmentDesign (UED), which seeks to generate environments tailored to the current learning agent to facilitate continuedlearning. However, UED is currently limited to generating levels, which are configurations of a specific task e.g.,the layout of a maze in a navigation task. We seek to improve state-of-the-art UED methods to generate entireenvironments, not just levels, which are novel tasks for a more general agent to train on and solve. Doing so willrequire more sophisticated generative AI techniques, where we look to leverage recent advances such as diffusionmethods. Ultimately, this work will operate on the intersection of multiple AI subfields, at a high level namely RLand generative AI.Alignment to EPSRC's strategies and research areasThe proposed work falls under the EPSRC's Artificial intelligence technologies remit. It aligns with EPSRC's goals ina number of ways, including developing new AI techniques that are deployable in real world situations. We will alsowork on the intersection of multiple AI subfields, in line with EPSRC's goal of supporting interdisciplinary researchmethods.