Associative Memory Based Experience Replay for Deep Reinforcement Learning
Associative Memory Based Experience Replay for Deep Reinforcement Learning
复制标题
DOI：
10.1145/3508352.3549387
复制
发表时间：
2022-07
期刊：
影响因子：
--
通讯作者：
Mengyuan Li;A. Kazemi;Ann Franchesca Laguna;Sharon Hu
中科院分区：
文献类型：
--
作者：
Mengyuan Li;A. Kazemi;Ann Franchesca Laguna;Sharon Hu
经验回放是深度强化学习（DRL）中的一个重要组成部分，它存储经验并生成经验供智能体真实的学习。最近，优先经验重放（PER）已被证明是强大的，并广泛部署在DRL代理。然而，在传统CPU或GPU架构上实现PER由于其频繁且不规则的存储器访问而招致显著的延迟开销。本文提出了一种软硬件协同设计的方法来设计一个基于联想记忆（AM）的PER，AMPER，与AM友好的优先级采样操作。AMPER在保持学习性能的同时，取代了PER中广泛使用的时间开销大的基于树遍历的优先级采样。此外，我们设计了一个基于AM的内存计算硬件架构，通过利用并行内存搜索操作来支持AMPER。与在GPU上运行的最先进PER相比，在所提出的硬件上运行时，AMPER显示出相当的学习性能，同时实现了55倍至270倍的延迟改善。
Experience replay is an essential component in deep reinforcement learning (DRL), which stores the experiences and generates experiences for the agent to learn in real time. Recently, prioritized experience replay (PER) has been proven to be powerful and widely deployed in DRL agents. However, implementing PER on traditional CPU or GPU architectures incurs significant latency overhead due to its frequent and irregular memory accesses. This paper proposes a hardware-software co-design approach to design an associative memory (AM) based PER, AMPER, with an AM-friendly priority sampling operation. AMPER replaces the widely-used time-costly tree-traversal-based priority sampling in PER while preserving the learning performance. Further, we design an in-memory computing hardware architecture based on AM to support AMPER by leveraging parallel in-memory search operations. AMPER shows comparable learning performance while achieving 55× to 270× latency improvement when running on the proposed hardware compared to the state-of-the-art PER running on GPU.