Towards Efficient Multi-Agent Learning Systems
Towards Efficient Multi-Agent Learning Systems
复制标题
DOI：
10.48550/arxiv.2305.13411
复制
发表时间：
2023-05
期刊：
影响因子：
--
通讯作者：
Kailash Gogineni;Peng Wei;Tian Lan;Guru Venkataramani
中科院分区：
文献类型：
--
作者：
Kailash Gogineni;Peng Wei;Tian Lan;Guru Venkataramani
多智能体强化学习（MARL）是一个越来越重要的研究领域，可以建模和控制多个大规模的自治系统。尽管其成就，现有的多智能体学习方法通常涉及昂贵的计算方面的训练时间和功率所产生的大的观察动作空间和大量的训练步骤。因此，一个关键的挑战是理解和表征几种流行的MARL算法在训练阶段的计算密集型功能。我们的初步实验揭示了对MARL算法关键模块的新见解，这些模块限制了MARL在现实系统中的采用。我们探索邻居采样策略，以提高缓存的局部性，并观察性能提高范围从26.66%（3代理）到27.39%（12代理）在计算密集型小批量采样阶段。此外，我们证明，提高局部性导致端到端的训练时间减少10.2%（12代理）相比，现有的多代理算法没有显着退化的平均奖励。
Multi-Agent Reinforcement Learning (MARL) is an increasingly important research field that can model and control multiple large-scale autonomous systems. Despite its achievements, existing multi-agent learning methods typically involve expensive computations in terms of training time and power arising from large observation-action space and a huge number of training steps. Therefore, a key challenge is understanding and characterizing the computationally intensive functions in several popular classes of MARL algorithms during their training phases. Our preliminary experiments reveal new insights into the key modules of MARL algorithms that limit the adoption of MARL in real-world systems. We explore neighbor sampling strategy to improve cache locality and observe performance improvement ranging from 26.66% (3 agents) to 27.39% (12 agents) during the computationally intensive mini-batch sampling phase. Additionally, we demonstrate that improving the locality leads to an end-to-end training time reduction of 10.2% (for 12 agents) compared to existing multi-agent algorithms without significant degradation in the mean reward.