Delay-Optimal Traffic Engineering through Multi-agent Reinforcement Learning
Delay-Optimal Traffic Engineering through Multi-agent Reinforcement Learning
复制标题
通过多智能体强化学习进行延迟优化流量工程
DOI：
10.1109/infcomw.2019.8845154
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
Pu Wang
中科院分区：
文献类型：
--
作者：
Pinyarash Pinyoanuntapong;Minwoo Lee;Pu Wang
交通工程是通过设计最佳转发和路由规则以满足大量交通流量的服务质量（QoS）要求来优化网络性能的最重要方法之一。端到端（E2E）延迟是关键指标之一。但是，由于网络不确定性和动态，优化E2E延迟在大规模多台网络中非常具有挑战性。本文提出了一个无模型的TE框架，该框架采用多代理增强学习以进行分布式控制以最大程度地减少E2E延迟。特别是，分布式TE被称为马尔可夫决策过程（MA-MDP）的多代理扩展。为了解决这个问题，提出了一个模块化和可组合的学习框架，该框架由三个交织模块组成，包括政策评估，政策改进和政策执行。每个组件都可以使用不同的算法及其扩展来实现。仿真结果表明，在高流量负载案例下，几种扩展的组合，例如双重学习，预期的政策评估和跨政策学习，可以提供出色的E2E延迟性能。
Traffic engineering is one of the most important methods of optimizing network performance by designing optimal forwarding and routing rules to meet the quality of service (QoS) requirements for a large volume of traffic flows. End-to-end (E2E) delay is one of the key TE metrics. Optimizing E2E delay, however, is very challenging in large-scale multihop networks due to the profound network uncertainties and dynamics. This paper proposes a model-free TE framework that adopts multi-agent reinforcement learning for distributed control to minimize the E2E delay. In particular, distributed TE is formulated as a multi-agent extension of Markov decision process (MA-MDP). To solve this problem, a modular and composable learning framework is proposed, which consists of three interleaving modules including policy evaluation, policy improvement, and policy execution. Each of component can be implemented using different algorithms along with their extensions. Simulation results show that the combination of several extensions, such as double learning, expected policy evaluation, and on-policy learning, can provide superior E2E delay performance under high traffic load cases.
DOI：
10.1109/infocom.2018.8485853
发表时间：
2018-01
期刊：
IEEE INFOCOM 2018 - IEEE Conference on Computer Communications
影响因子：
--
作者：
Zhiyuan Xu;Jian Tang;Jingsong Meng;Weiyi Zhang;Yanzhi Wang;C. Liu;Dejun Yang
通讯作者：
Zhiyuan Xu;Jian Tang;Jingsong Meng;Weiyi Zhang;Yanzhi Wang;C. Liu;Dejun Yang