Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings
Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings
复制标题
DOI：
--
复制
发表时间：
2021-03
期刊：
影响因子：
--
通讯作者：
Lili Chen;Kimin Lee;A. Srinivas;P. Abbeel
中科院分区：
文献类型：
--
作者：
Lili Chen;Kimin Lee;A. Srinivas;P. Abbeel
非政策性深度强化学习（RL）的最新进展使我们在通过视觉观察完成复杂任务方面取得了令人瞩目的成功。经验重放通过重复使用过去的经验提高了样本效率，而卷积神经网络（CNN）则能有效处理高维输入。然而，这些技术需要很高的内存和计算带宽。在本文中，我们提出了 "高效强化学习的存储嵌入"（SEER），它是对现有非策略 RL 方法的一种简单修改，以满足这些计算和内存要求。为了减少 CNN 中梯度更新的计算开销，我们在训练早期冻结了 CNN 编码器的下层，因为它们的参数会提前收敛。此外，我们还通过存储用于经验重放的低维潜在向量而不是高维图像来降低内存要求，从而实现重放缓冲区容量的自适应增加，这在内存受限的环境中是一项有用的技术。在实验中，我们发现 SEER 不会降低 RL 代理的性能，同时在各种 DeepMind 控制环境和 Atari 游戏中大大节省了计算量和内存。
Recent advances in off-policy deep reinforcement learning (RL) have led to impressive success in complex tasks from visual observations. Experience replay improves sample-efficiency by reusing experiences from the past, and convolutional neural networks (CNNs) process high-dimensional inputs effectively. However, such techniques demand high memory and computational bandwidth. In this paper, we present Stored Embeddings for Efficient Reinforcement Learning (SEER), a simple modification of existing off-policy RL methods, to address these computational and memory requirements. To reduce the computational overhead of gradient updates in CNNs, we freeze the lower layers of CNN encoders early in training due to early convergence of their parameters. Additionally, we reduce memory requirements by storing the low-dimensional latent vectors for experience replay instead of high-dimensional images, enabling an adaptive increase in the replay buffer capacity, a useful technique in constrained-memory settings. In our experiments, we show that SEER does not degrade the performance of RL agents while significantly saving computation and memory across a diverse set of DeepMind Control environments and Atari games.