On Centralized Critics in Multi-Agent Reinforcement Learning
On Centralized Critics in Multi-Agent Reinforcement Learning
复制标题
DOI：
10.1613/jair.1.14386
复制
发表时间：
2023-05
期刊：
影响因子：
--
通讯作者：
Xueguang Lyu;Andrea Baisero;Yuchen Xiao;Brett Daley;Chris Amato
中科院分区：
文献类型：
--
作者：
Xueguang Lyu;Andrea Baisero;Yuchen Xiao;Brett Daley;Chris Amato
分散式执行的集中式训练，即代理以集中的方式离线训练，并以分散的方式在线执行，已成为多代理强化学习（MARL）中的一种流行方法。特别是，开发参与者-批评者方法已经变得流行，这种方法可以用集中的批评者来训练分散的参与者，其中集中的批评者可以访问整个系统的全局信息，包括真实的系统状态。这种集中式批评在提供离线信息的情况下是可能的，而不是用于在线执行。虽然这些方法在许多领域表现良好，并已成为MARL的事实上的标准，但在这种情况下使用集中式批评尚未得到充分的理论或经验分析。因此，在本文中，我们正式分析了集中式和分散式批评方法，并分析了在部分可观察的环境中使用基于状态的批评的效果。我们得出了与普遍直觉相反的理论：批评家集中化并不严格有益，使用国家价值可能有害。我们进一步证明，与基于历史的批评相比，基于国家的批评尤其会引入意想不到的偏见和方差。最后，我们通过在广泛的常见多智能体基准上比较不同形式的批评来证明该理论如何应用于实践。实验显示了实际问题，如部分可观察性表征学习的困难，这突出了为什么理论问题在文献中经常被忽视。
Centralized Training for Decentralized Execution, where agents are trained offline in a centralized fashion and execute online in a decentralized manner, has become a popular approach in Multi-Agent Reinforcement Learning (MARL). In particular, it has become popular to develop actor-critic methods that train decentralized actors with a centralized critic where the centralized critic is allowed access global information of the entire system, including the true system state. Such centralized critics are possible given offline information and are not used for online execution. While these methods perform well in a number of domains and have become a de facto standard in MARL, using a centralized critic in this context has yet to be sufficiently analyzed theoretically or empirically. In this paper, we therefore formally analyze centralized and decentralized critic approaches, and analyze the effect of using state-based critics in partially observable environments. We derive theories contrary to the common intuition: critic centralization is not strictly beneficial, and using state values can be harmful. We further prove that, in particular, state-based critics can introduce unexpected bias and variance compared to history-based critics. Finally, we demonstrate how the theory applies in practice by comparing different forms of critics on a wide range of common multi-agent benchmarks. The experiments show practical issues such as the difficulty of representation learning with partial observability, which highlights why the theoretical problems are often overlooked in the literature.