The Utility of Sparse Representations for Control in Reinforcement Learning
The Utility of Sparse Representations for Control in Reinforcement Learning
复制标题
DOI：
10.1609/aaai.v33i01.33014384
复制
发表时间：
2018-11
期刊：
影响因子：
--
通讯作者：
Vincent Liu;Raksha Kumaraswamy;Lei Le;Martha White
中科院分区：
文献类型：
--
作者：
Vincent Liu;Raksha Kumaraswamy;Lei Le;Martha White
我们研究了强化学习中控制的稀疏表示。虽然这些表示广泛用于计算机视觉，但它们在强化学习中的流行仅限于稀疏编码，其中提取新数据的表示可能是计算密集型的。在这里，我们开始证明，学习控制策略的增量表示从一个标准的神经网络在经典的控制域失败，而学习表示从一个神经网络，具有稀疏性的属性是有效的。我们提供的证据表明，这是因为稀疏表示提供了局部性，因此避免了灾难性的干扰，特别是保持一致，稳定的值进行自举。然后我们讨论如何学习这种稀疏表示。我们探讨了分布正则化的想法，其中隐藏节点的激活被鼓励以匹配特定的分布，从而在整个时间内产生稀疏的激活。我们确定了一种简单但有效的方法来获得稀疏表示，这是以前提出的策略所不能提供的，这使得进一步研究强化学习的稀疏表示更加实用。
We investigate sparse representations for control in reinforcement learning. While these representations are widely used in computer vision, their prevalence in reinforcement learning is limited to sparse coding where extracting representations for new data can be computationally intensive. Here, we begin by demonstrating that learning a control policy incrementally with a representation from a standard neural network fails in classic control domains, whereas learning with a representation obtained from a neural network that has sparsity properties enforced is effective. We provide evidence that the reason for this is that the sparse representation provides locality, and so avoids catastrophic interference, and particularly keeps consistent, stable values for bootstrapping. We then discuss how to learn such sparse representations. We explore the idea of Distributional Regularizers, where the activation of hidden nodes is encouraged to match a particular distribution that results in sparse activation across time. We identify a simple but effective way to obtain sparse representations, not afforded by previously proposed strategies, making it more practical for further investigation into sparse representations for reinforcement learning.