Towards Open-ended Reinforcement Learning using Synthetic Environment Generation

点击翻译按钮获取中文摘要

Sequential decision making problems are ubiquitous in engineering and science. Reinforcement Learning (RL), anAI paradigm where agents learn decision-making skills via trial-and-error interactions with their environment, hasachieved significant success in handling complex decision tasks. However, these agents often struggle to generalize,exhibiting suboptimal performance on previously unseen tasks. Furthermore, once deployed, an AI that underperforms in a new task often lacks opportunities for improvement, as its learning process ceases after the initialtraining phase. These challenges restrict the practical use of such systems in real-worlds settings, such as sim2real.Open-ended Learning (OEL) seeks to overcome these limitations with the goal of producing learning systems thatare robust to situations not explicitly considered during design and training.Aims and ObjectivesWhile there are many possible directions towards achieving OEL agents, this proposal specifically focuses on thedevelopment of automated curricula methods through novel techniques for synthetic environment generation. Assuch, the goals of this proposal are decomposed as follows:Develop new methods for generating synthetic tasks/environments.Use synthetic environment generation to develop novel Unsupervised Environment Design (UED) methods forautomatic environment curricula generation.Show that applying such curricula to agent training produces agents with strong out-of-distribution generalization.Novelty of the Research MethodologyThe proposed research primarily looks to extend work in the developing subfield of Unsupervised EnvironmentDesign (UED), which seeks to generate environments tailored to the current learning agent to facilitate continuedlearning. However, UED is currently limited to generating levels, which are configurations of a specific task e.g.,the layout of a maze in a navigation task. We seek to improve state-of-the-art UED methods to generate entireenvironments, not just levels, which are novel tasks for a more general agent to train on and solve. Doing so willrequire more sophisticated generative AI techniques, where we look to leverage recent advances such as diffusionmethods. Ultimately, this work will operate on the intersection of multiple AI subfields, at a high level namely RLand generative AI.Alignment to EPSRC's strategies and research areasThe proposed work falls under the EPSRC's Artificial intelligence technologies remit. It aligns with EPSRC's goals ina number of ways, including developing new AI techniques that are deployable in real world situations. We will alsowork on the intersection of multiple AI subfields, in line with EPSRC's goal of supporting interdisciplinary researchmethods.