Taming Communication and Sample Complexities in Decentralized Policy Evaluation for Cooperative Multi-Agent Reinforcement Learning-猫眼课题宝

DOI：

发表时间：

2021

期刊：

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Xin Zhang;Zhuqing Liu;Jia Liu;Zhengyuan Zhu-;Songtao Lu

协作多智能体强化学习（MARL）近年来受到越来越多的关注，并已发现许多科学和工程应用。然而，从许多合作MARL算法设计（例如，行动者-批评者框架）是政策评价问题，只能以分散的方式进行。在本文中，我们专注于分散的MARL政策评估与非线性函数逼近，这是经常看到的深MARL。我们首先证明了经验分散MARL政策评估问题可以转化为分散非凸-强-凹极大极小鞍点问题。然后，我们开发了一个分散的基于梯度的下降上升算法称为GT-GDA，享有O（1 /T）的收敛速度。为了进一步降低样本复杂度，我们提出了两个分散随机优化算法GT-SRVR和GT-SRVR I，它们通过方差缩减技术增强了GT-GDA。我们证明了所有的算法都具有O（1 /T）的收敛速度到一个稳定点的重新minimax问题。此外，GT-SRVR和GT-SRVR I的快速收敛速度意味着O（（cid：15）− 2）通信复杂度和O（m <$n（cid：15）− 2）样本复杂度，其中m是代理的数量，n是

Cooperative multi-agent reinforcement learning (MARL) has received increasing attention in recent years and has found many scientiﬁc and engineering applications. However, a key challenge arising from many cooperative MARL algorithm designs (e.g., the actor-critic framework) is the policy evaluation problem, which can only be conducted in a decentralized fashion. In this paper, we focus on decentralized MARL policy evaluation with nonlinear function approximation, which is often seen in deep MARL. We ﬁrst show that the empirical decentralized MARL policy evaluation problem can be reformulated as a decentralized nonconvex-strongly-concave minimax saddle point problem. We then develop a decentralized gradient-based descent ascent algorithm called GT-GDA that enjoys a convergence rate of O (1 /T ) . To further reduce the sample complexity, we pro-pose two decentralized stochastic optimization algorithms called GT-SRVR and GT-SRVR I , which enhance GT-GDA by variance reduction techniques. We show that all algorithms all enjoy an O (1 /T ) convergence rate to a stationary point of the reformulated minimax problem. Moreover, the fast convergence rates of GT-SRVR and GT-SRVR I imply O ( (cid:15) − 2 ) communication complexity and O ( m √ n(cid:15) − 2 ) sample complexity, where m is the number of agents and n is

Taming Communication and Sample Complexities in Decentralized Policy Evaluation for Cooperative Multi-Agent Reinforcement Learning

文献摘要

相似文献