IHG-MA: Inductive heterogeneous graph multi-agent reinforcement learning for multi-intersection traffic signal control
IHG-MA: Inductive heterogeneous graph multi-agent reinforcement learning for multi-intersection traffic signal control
复制标题
DOI：
10.1016/j.neunet.2021.03.015
复制
发表时间：
2021-03
期刊：
影响因子：
--
通讯作者：
Shantian Yang;Bo Yang;Zhongfeng Kang;Lihui Deng
中科院分区：
文献类型：
--
作者：
Shantian Yang;Bo Yang;Zhongfeng Kang;Lihui Deng
多智能体深度强化学习（MDRL）在多路口交通信号控制中得到了广泛应用。MDRL算法在特定的交通网络中通过专门的多智能体设置产生分散的协同交通信号策略。然而，最先进的MDRL算法似乎有一些缺点。(1)期望交通信号策略能够在不同的交通网络中顺利迁移，但采用的专用多智能体设置阻碍了交通信号策略在新交通网络中的迁移和推广。(2)现有的基于深度神经网络的MDRL算法不能灵活地处理穿越交通网络的时变车辆数量。(3)现有的基于同构图神经网络的MDRL算法无法捕获交通网络中对象的异构特征。基于以上观察结果，本文提出了一种用于多路口交通信号控制的算法，称为归纳异构图多智能体Actor-critic （IHG-MA）算法。本文提出的IHG- ma算法有两个特点：(1)使用一种归纳算法，即提出的归纳异构图神经网络（IHG）进行表示学习。提出的IHG算法可以为以前未见过的节点（例如，新进入车辆）和新图（例如，新交通网络）生成嵌入。但与基于同构图神经网络的算法不同，IHG算法不仅编码每个节点的异构特征，而且编码异构的结构（图）信息。(2)它还使用提出的多智能体参与者-批评者（MA）进行策略学习，这是一个分散的合作框架。本文提出的MA框架利用最后的嵌入来计算q值和策略，然后通过q值和策略损失来优化整个算法。在不同交通数据集上的实验结果表明，IHG-MA算法在多交通指标方面优于现有算法，是一种很有前途的多路口交通信号控制新算法。
Multi-agent deep reinforcement learning (MDRL) has been widely applied in multi-intersection traffic signal control. The MDRL algorithms produce the decentralized cooperative traffic-signal policies via specialized multi-agent settings in certain traffic networks. However, the state-of-the-art MDRL algorithms seem to have some drawbacks. (1) It is desirable that the traffic-signal policies can be smoothly transferred to diverse traffic networks, however, the adopted specialized multi-agent settings hinder the traffic-signal policies to transfer and generalize to new traffic networks. (2) Existing MDRL algorithms which are based on deep neural networks cannot flexibly tackle a time-varying number of vehicles traversing the traffic networks. (3) Existing MDRL algorithms which are based on homogeneous graph neural networks fail to capture the heterogeneous features of objects in traffic networks. Motivated by the above observations, in this paper, we propose an algorithm, referred to as Inductive Heterogeneous Graph Multi-agent Actor–critic (IHG-MA) algorithm, for multi-intersection traffic signal control. The proposed IHG-MA algorithm has two features: (1) It conducts representation learning using a proposed inductive heterogeneous graph neural network (IHG), which is an inductive algorithm. The proposed IHG algorithm can generate embeddings for previously unseen nodes (e.g., new entry vehicles) and new graphs (e.g., new traffic networks). But unlike the algorithms based on the homogeneous graph neural network, IHG algorithm not only encodes heterogeneous features of each node, but also encodes heterogeneous structural (graph) information. (2) It also conducts policy learning using a proposed multi-agent actor–critic(MA), which is a decentralized cooperative framework. The proposed MA framework employs the final embeddings to compute theQ-value and policy, and then optimizes the whole algorithm via theQ-value and policy loss. Experimental results on different traffic datasets illustrate that IHG-MA algorithm outperforms the state-of-the-art algorithms in terms of multiple traffic metrics, which seems to be a new promising algorithm for multi-intersection traffic signal control.