Scaling Unsupervised Environment Design

强化学习（RL）是机器学习的一个子领域，其中智能体（例如自动驾驶汽车）从环境（例如真实道路/模拟道路）中的行为中学习。尽管在解决复杂的电子游戏（雅达利、围棋、星际争霸）方面取得了很大进展，但它还没有成功地应用于许多现实世界的问题。造成这种情况的根本原因是RL代理无法归纳到看不见的场景。具体来说，由于不可避免的模拟不准确性，在模拟中训练的RL代理在部署到现实世界中时不能很好地迁移（注意，由于需要大量的训练数据和潜在的危险，在现实世界中训练代理通常是不切实际的）。最近的开创性工作通过培训老师来学习提出高质量的场景（例如道路布局）供智能体进行训练，证明了对泛化的显著经验效益，反映了监督学习的结果，显示了数据质量在泛化中的重要性。这项工作的一个限制是，教师必须从稀疏和有噪声的信号中学习，导致样本效率低，需要大量的计算资源，这意味着它只成功地应用于非常简单的问题。为了减少信号噪声，我提出了一些方法，鼓励教师使用近似惊喜、易辨别性和学习潜在空间中的距离等指标来维持一组不同的场景。此外，我提出了一种新的数据增强方法，即将场景分解为一组“子场景”，以最小的计算成本扩展训练数据。最后，目前最先进的方法是通过应用随机扰动来训练教师。我建议一种有针对性的扰动方法，通过不断逼近代理的遗憾（它在任务中的表现与最优代理的表现之间的差异），并在最低的地方应用扰动。所有这些技术都旨在提高整个过程的效率，减少所需的资源，并允许将这种强大的技术开放到更复杂的领域，从而使自动驾驶等现实世界的应用受益。值得注意的是，虽然我使用自动驾驶作为运行示例，但正在开发的方法将适用于任何强化学习问题，并将在各种环境中进行评估。该项目属于EPSRC人工智能技术研究领域。

Reinforcement learning (RL) is a subfield of machine learning where an agent (e.g. an autonomous vehicle) learns from acting in an environment (e.g. a real road / simulation of a road). Despite making great progress in solving complex video games (Atari, Go, StarCraft), it has not yet been successfully applied to many real world problems. The root cause of this is the inability of RL agents to generalise to unseen scenarios. Specifically, an RL agent trained in a simulation doesn't transfer well when deployed in the real world, due to the inevitable inaccuracies of simulation (note that, due to the large volume of training data needed and potential dangers, it is often impractical to train an agent in the real world).Recent pioneering work has demonstrated significant empirical benefits to generalisation by training a teacher that learns to propose high-quality scenarios (e.g. road layouts) for the agent to train on, mirroring results from supervised learning that have shown the importance of data quality in generalisation. A limitation to this work is that the teacher has to learn from a sparse and noisy signal, resulting in low sample efficiency and necessitating large computational resources, meaning it has only been successfully applied to very simple problems. To reduce signal noise, I have proposed methods encouraging the teacher to maintain a diverse set of scenarios using metrics for approximated surprise, ease of discrimination and distance in a learned latent space. Furthermore, I propose a novel data augmentation method, whereby scenarios are decomposed into a set of 'sub-scenarios', expanding the training data with minimal computational cost. Finally, the current state of the art method trains the teacher by applying random perturbations. I suggest a method for targeted perturbations by constantly approximating the agent's regret (the difference between how well it did at the task and how well an optimal agent would have done) and applying perturbations where this is lowest. All these techniques aim to improve the efficiency of the overall process, reducing the resources needed and allowing this powerful technique to be opened up to more complex domains, benefiting real world applications like autonomous driving. It should be noted that, while I use autonomous driving as a running example, the methods being developed will be generalisable to any RL problem and will be evaluated over a diverse range of environments.This project falls within the EPSRC Artificial intelligence technologies research area.