Reinforcement Learning in Latent Action Sequence Space
Reinforcement Learning in Latent Action Sequence Space
复制标题
潜在动作序列空间中的强化学习
DOI：
--
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
H. Yamakawa
中科院分区：
文献类型：
--
作者：
Heecheol Kim;Masanori Yamada;Kosuke Miyoshi;Tomoharu Iwata;H. Yamakawa
强化学习在实际应用中的一个问题是动作搜索空间的高维性，这来自于动作随时间的组合。为了降低动作序列搜索空间的维数，研究了宏动作，宏动作是解决任务的原始动作序列。然而，之前的研究依赖于人类来定义宏动作或假设宏动作是相同原始动作的重复。我们提出了编码动作序列强化学习（EASRL），这是一种强化学习方法，可以在高维动作序列搜索空间的潜在空间中学习灵活的动作序列。使用EASRL，编码器和解码器网络通过使用变分自编码器将宏动作映射到潜在空间中来训练演示数据。然后，我们在潜在空间中学习策略网络，它是给定状态的编码宏动作的分布。通过在潜在空间中学习，我们可以降低动作序列搜索空间的维数，并处理各种模式的动作序列。我们通过实验证明，该方法在需要大量搜索的任务上优于其他强化学习方法。
One problem in real-world applications of reinforcement learning is the high dimensionality of the action search spaces, which comes from the combination of actions over time. To reduce the dimensionality of action sequence search spaces, macro actions have been studied, which are sequences of primitive actions to solve tasks. However, previous studies relied on humans to define macro actions or assumed macro actions to be repetitions of the same primitive actions. We propose encoded action sequence reinforcement learning (EASRL), a reinforcement learning method that learns flexible sequences of actions in a latent space for a high-dimensional action sequence search space. With EASRL, encoder and decoder networks are trained with demonstration data by using variational autoencoders for mapping macro actions into the latent space. Then, we learn a policy network in the latent space, which is a distribution over encoded macro actions given a state. By learning in the latent space, we can reduce the dimensionality of the action sequence search space and handle various patterns of action sequences. We experimentally demonstrate that the proposed method outperforms other reinforcement learning methods on tasks that require an extensive amount of search.