A semi-decentralized feudal multi-agent learned-goal algorithm for multi-intersection traffic signal control
A semi-decentralized feudal multi-agent learned-goal algorithm for multi-intersection traffic signal control
复制标题
DOI：
10.1016/j.knosys.2020.106708
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Shantian Yang;Bo Yang
中科院分区：
文献类型：
--
作者：
Shantian Yang;Bo Yang
交通信号控制有助于减少交通拥堵，因此已经研究了几十年。采用分层深度强化学习的多路口交通信号控制算法已被证明具有最先进的性能。对于这些方法，通常采用两层层次结构，其中低级策略采用由高级策略诱导的潜在目标。现有研究要么人工设计潜在目标，要么从环境中获取潜在目标。然而，这些潜在目标并不是最优的，这就导致了低级策略的非最优。为了改进这一点，我们提出了一种学习目标软行为者批评家（LSAC）算法，该算法自动学习最优潜在目标，然后将其用于低级策略。在此基础上，我们提出了一种半分散的封建多智能体（SFM）框架，该框架可以缓解现有多智能体框架所面临的状态空间随着智能体数量的增加而快速增长的问题。结合上述两种方法，本文提出了一种多交叉口交通信号控制的SFM-LSAC算法。在SFM-LSAC算法中还引入了注意机制等其他技术。三种实际交通场景的实验结果表明，SFM-LSAC算法优于其他四种最先进的多交叉口交通信号控制算法，即降低了平均交叉口延迟、平均队列长度、平均行驶时间，同时提高了平均流量和平均行驶速度，是一种很有前景的多交叉口交通信号控制新算法。
Traffic signal control helps to reduce traffic congestion and thus has been studied for a few decades. Algorithms for multi-intersection traffic signal control that adopt hierarchical deep reinforcement learning have been shown to achieve state-of-the-art performance. For these methods, a two-level hierarchical structure is generally used, in which the low-level policies employ latent goals induced by the high-level policies. Existing research either designs the latent goals manually or acquires the latent goals from the environment. However, these latent goals are not optimal, which leads to the non-optimal low-level policies. To improve this, we propose a learned-goal soft actor–critic (LSAC) algorithm, by which the optimal latent goals are automatically learned and then are used in low-level policies. We then propose a semi-decentralized feudal multi-agent (SFM) framework, which can alleviate the problem that existing multi-agent framework faces, i.e., the fast growing state space with the increase of the number of agents. Combining the above two proposed methods, an SFM-LSAC algorithm is proposed in this paper for multi-intersection traffic signal control. Other techniques are also incorporated into the SFM-LSAC algorithm such as attention mechanism. Experimental results in three real-world traffic scenarios illustrate that the SFM-LSAC algorithm outperforms other four state-of-the-art multi-intersection traffic signal control algorithms, i.e., reduces the average intersection delay, the average queue length, the average travel time, meanwhile improves the average flow and average travel speed, thus could be a new promising algorithm for multi-intersection traffic signal control.