Towards Open-ended Reinforcement Learning using Synthetic Environment Generation

序贯决策问题在工程和科学中普遍存在。强化学习(RL)是一种人工智能范式，它通过与环境的反复交互来学习决策技能，在处理复杂的决策任务方面取得了巨大的成功。然而，这些代理通常难以推广，在以前未见过的任务中表现出次优性能。此外，一旦部署，在新任务中表现不佳的人工智能往往缺乏改进的机会，因为它的学习过程在初始训练阶段后停止。这些挑战限制了此类系统在真实世界环境中的实际应用，例如Sim2 Real.开放式学习(OEL)试图克服这些限制，目标是生成对设计和培训中未明确考虑的情况具有健壮性的学习系统。目的和目标虽然有许多可能的方向来实现OEL代理，但本提案特别关注通过合成环境生成的新技术来开发自动化课程方法。因此，本建议的目标分解为：开发生成合成任务/环境的新方法。使用合成环境生成开发用于自动环境课程生成的新的无监督环境设计(UED)方法。表明将此类课程应用于代理培训产生的代理具有很强的非分布泛化能力。研究方法论的新奇之处本研究主要寻求扩展无监督环境设计(UED)发展中的子领域的工作，该领域寻求生成适合当前学习代理的环境以促进继续学习。然而，UED目前仅限于生成级别，其是特定任务的配置，例如导航任务中迷宫的布局。我们寻求改进最先进的UED方法，以生成整个环境，而不仅仅是级别，这是更一般的代理需要培训和解决的新任务。要做到这一点，将需要更复杂的生成性人工智能技术，在这方面，我们希望利用最近的进展，如扩散方法。最终，这项工作将在多个人工智能子领域的交叉点上进行，即高水平的生成性人工智能。与EPSRC的战略和研究领域保持一致这项拟议的工作属于EPSRC的人工智能技术范围。它在许多方面与EPSRC的目标一致，包括开发可在现实世界中部署的新人工智能技术。我们还将致力于多个人工智能子领域的交叉，以符合EPSRC支持跨学科研究方法的目标。

Sequential decision making problems are ubiquitous in engineering and science. Reinforcement Learning (RL), anAI paradigm where agents learn decision-making skills via trial-and-error interactions with their environment, hasachieved significant success in handling complex decision tasks. However, these agents often struggle to generalize,exhibiting suboptimal performance on previously unseen tasks. Furthermore, once deployed, an AI that underperforms in a new task often lacks opportunities for improvement, as its learning process ceases after the initialtraining phase. These challenges restrict the practical use of such systems in real-worlds settings, such as sim2real.Open-ended Learning (OEL) seeks to overcome these limitations with the goal of producing learning systems thatare robust to situations not explicitly considered during design and training.Aims and ObjectivesWhile there are many possible directions towards achieving OEL agents, this proposal specifically focuses on thedevelopment of automated curricula methods through novel techniques for synthetic environment generation. Assuch, the goals of this proposal are decomposed as follows:Develop new methods for generating synthetic tasks/environments.Use synthetic environment generation to develop novel Unsupervised Environment Design (UED) methods forautomatic environment curricula generation.Show that applying such curricula to agent training produces agents with strong out-of-distribution generalization.Novelty of the Research MethodologyThe proposed research primarily looks to extend work in the developing subfield of Unsupervised EnvironmentDesign (UED), which seeks to generate environments tailored to the current learning agent to facilitate continuedlearning. However, UED is currently limited to generating levels, which are configurations of a specific task e.g.,the layout of a maze in a navigation task. We seek to improve state-of-the-art UED methods to generate entireenvironments, not just levels, which are novel tasks for a more general agent to train on and solve. Doing so willrequire more sophisticated generative AI techniques, where we look to leverage recent advances such as diffusionmethods. Ultimately, this work will operate on the intersection of multiple AI subfields, at a high level namely RLand generative AI.Alignment to EPSRC's strategies and research areasThe proposed work falls under the EPSRC's Artificial intelligence technologies remit. It aligns with EPSRC's goals ina number of ways, including developing new AI techniques that are deployable in real world situations. We will alsowork on the intersection of multiple AI subfields, in line with EPSRC's goal of supporting interdisciplinary researchmethods.