AccMER: Accelerating Multi-Agent Experience Replay with Cache Locality-Aware Prioritization
AccMER: Accelerating Multi-Agent Experience Replay with Cache Locality-Aware Prioritization
复制标题
DOI：
10.1109/asap57973.2023.00041
复制
发表时间：
2023-05
期刊：
影响因子：
--
通讯作者：
Kailash Gogineni;Yongsheng Mei;Peng Wei;Tian Lan;Guru Venkataramani
中科院分区：
文献类型：
--
作者：
Kailash Gogineni;Yongsheng Mei;Peng Wei;Tian Lan;Guru Venkataramani
多智能体经验回放（MER）是离线强化学习（RL）算法的一个关键组成部分。通过记忆和重用过去的经验，经验重放显着提高了RL算法的稳定性和学习效率。在集中训练和分散执行（CTDE）范式下，在许多情况下，多个智能体在一个共享的环境中进行交互。现有的多智能体强化学习（MARL）算法在采样阶段采用均匀采样或基于优先权的经验回放来提高迁移数据的采样效率。然而，通过处理器存储器层次结构移动每个代理的转换数据历史是性能限制器。此外，由于代理的转换在每次迭代中不断更新，有限的高速缓存容量导致高速缓存未命中增加。为此，我们建议AccMER，重复使用的过渡（经验）为窗口的$n$步骤，以提高该高速缓存的本地性和最大限度地减少过渡数据的移动，而不是在每一步采样新的过渡。具体来说，我们的优化使用优先级权重来选择转换，以便只有高优先级的转换将被频繁重用，从而提高该高速缓存的性能。我们在捕食者-猎物环境中的实验结果证明了基于优先权重用基本转换的有效性，与现有的优先MER算法相比，我们观察到端到端训练时间减少了25.4%（对于32个代理），而平均奖励没有显着下降。
Multi-Agent Experience Replay (MER) is a key component of off-policy reinforcement learning (RL) algorithms. By remembering and reusing experiences from the past, experience replay significantly improves the stability of RL algorithms and their learning efficiency. In many scenarios, multiple agents interact in a shared environment during online training under centralized training and decentralized execution (CTDE) paradigm. Current multi-agent reinforcement learning (MARL) algorithms consider experience replay with uniform sampling or based on priority weights to improve transition data sample efficiency in the sampling phase. However, moving transition data histories for each agent through the processor memory hierarchy is a performance limiter. Also, as the agents' transitions continuously renew every iteration, the finite cache capacity results in increased cache misses. To this end, we propose AccMER, that repeatedly reuses the transitions (experiences) for a window of $n$ steps in order to improve the cache locality and minimize the transition data movement, instead of sampling new transitions at each step. Specifically, our optimization uses priority weights to select the transitions so that only high-priority transitions will be reused frequently, thereby improving the cache performance. Our experimental results on the Predator- Prey environment demonstrate the effectiveness of reusing the essential transitions based on the priority weights, where we observe an end-to-end training time reduction of 25.4% (for 32 agents) compared to existing prioritized MER algorithms without notable degradation in the mean reward.