Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
复制标题
DOI：
--
复制
发表时间：
2020-06
期刊：
影响因子：
--
通讯作者：
Roberta Raileanu;M. Goldstein;Denis Yarats;Ilya Kostrikov;R. Fergus
中科院分区：
文献类型：
--
作者：
Roberta Raileanu;M. Goldstein;Denis Yarats;Ilya Kostrikov;R. Fergus
深度强化学习（RL）代理通常无法推广到看不见的场景，即使它们在语义相似的环境中进行了训练。数据扩充最近被证明可以提高RL代理的样本效率和泛化能力。然而，不同的任务往往受益于不同类型的数据增强。在本文中，我们比较了三种方法自动找到一个适当的增强。这些与策略和值函数的两个新的正则化项相结合，需要使用数据增强理论上适合某些演员-评论家算法。我们评估我们的方法在Procgen基准测试，其中包括16个程序生成的环境，并表明它提高了测试性能的~40%相对于标准RL算法。我们的代理优于其他专门设计用于提高RL泛化能力的基线。此外，我们表明，我们的代理学习的政策和表示，更强大的环境中的变化，不影响代理，如背景。我们的实现可以在这个https URL上找到。
Deep reinforcement learning (RL) agents often fail to generalize to unseen scenarios, even when they are trained on many instances of semantically similar environments. Data augmentation has recently been shown to improve the sample efficiency and generalization of RL agents. However, different tasks tend to benefit from different kinds of data augmentation. In this paper, we compare three approaches for automatically finding an appropriate augmentation. These are combined with two novel regularization terms for the policy and value function, required to make the use of data augmentation theoretically sound for certain actor-critic algorithms. We evaluate our methods on the Procgen benchmark which consists of 16 procedurally-generated environments and show that it improves test performance by ~40% relative to standard RL algorithms. Our agent outperforms other baselines specifically designed to improve generalization in RL. In addition, we show that our agent learns policies and representations that are more robust to changes in the environment that do not affect the agent, such as the background. Our implementation is available at this https URL.