TD3lite: FPGA Acceleration of Reinforcement Learning with Structural and Representation Optimizations
TD3lite: FPGA Acceleration of Reinforcement Learning with Structural and Representation Optimizations
复制标题
DOI：
10.1109/fpl57034.2022.00023
复制
发表时间：
2022-08
期刊：
影响因子：
--
通讯作者：
Chan-Wei Hu;Jiangkun Hu;S. Khatri
中科院分区：
文献类型：
--
作者：
Chan-Wei Hu;Jiangkun Hu;S. Khatri
强化学习 (RL) 是一种有效且日益流行的优化和决策机器学习方法。然而，现代强化学习技术，例如深度 Q 学习，通常需要神经网络推理和训练，因此计算成本很高。例如，双延迟深度确定性策略梯度 (TD3) 是一种最先进的 RL 技术，使用多达 6 个神经网络。在这项工作中，我们研究了基于 FPGA 的 TD3 加速。为了解决 TD3 的多个神经网络的推理和训练带来的资源和计算开销，我们提出了 TD3lite，这是一种由网络共享技术与位宽优化的块浮点运算相结合组成的集成方法。 TD3lite 在多个具有连续状态和动作空间的机器人基准测试上进行评估。与 CPU 和 GPU 实现相比，TD3lite 的学习性能仅下降 5.7%，分别实现了 21 倍和 8 倍加速。其能源效率是 GPU 实现的 26 倍。此外，与 TD3 的传统单精度浮点表示相比，它使用的 FPGA 资源减少了约 25 - 40%。
Reinforcement learning (RL) is an effective and increasingly popular machine learning approach for optimization and decision-making. However, modern reinforcement learning techniques, such as deep Q-learning, often require neural network inference and training, and therefore are computationally expensive. For example, Twin-Delay Deep Deterministic Policy Gradient (TD3), a state-of-the-art RL technique, uses as many as 6 neural networks. In this work, we study the FPGA-based acceleration of TD3. To address the resource and computational overhead due to inference and training of the multiple neural networks of TD3, we propose TD3lite, an integrated approach consisting of a network sharing technique combined with bitwidth-optimized block floating-point arithmetic. TD3lite is evaluated on several robotic benchmarks with continuous state and action spaces. With only 5.7% learning performance degradation, TD3lite achieves 21 ×and 8 ×speedup compared to CPU and GPU implementations, respectively. Its energy efficiency is 26 ×of the GPU implementation. Moreover, it utilizes ~ 25 - 40% fewer FPGA resources compared to a conventional sinale-precision floating-point representation of TD3.