Interference Management for Cellular-Connected UAVs: A Deep Reinforcement Learning Approach
Interference Management for Cellular-Connected UAVs: A Deep Reinforcement Learning Approach
复制标题
DOI：
10.1109/twc.2019.2900035
复制
发表时间：
2019-04-01
影响因子：
10.4
通讯作者：
Bettstetter, Christian
中科院分区：
文献类型：
--
作者：
Challita, Ursula;Saad, Walid;Bettstetter, Christian
针对蜂窝互联无人机网络，提出了一种干扰感知的路径规划方案。具体地说，每架无人机的目标是在最大化能源效率和最小化无线延迟和对其路径沿线的地面网络造成的干扰之间实现权衡。这个问题被描述为无人机之间的动态博弈。为了解决这一博弈问题，提出了一种基于回声状态网络(ESN)神经元的深度强化学习算法。引入的深度ESN体系结构经过训练，允许每架无人机将网络状态的每个观测映射到一个动作，目标是最小化一系列与时间相关的效用函数。每架无人机使用ESN来学习其路径沿途不同位置的最优路径、发射功率和小区关联向量。证明了该算法在收敛时达到了子博弈完美纳什均衡。此外，还导出了无人机高度的上界和下界，从而降低了算法的计算复杂度。仿真结果表明，所提出的方案具有更好的每无人机无线时延和每地面用户速率(UE)，而所需的步骤与考虑通过最短距离向相应目的地移动的启发式基线相当。结果还表明，无人机的最佳高度根据地面网络密度和UE数据速率要求而变化，对于最小化地面UE的干扰水平以及无人机的无线传输延迟起着至关重要的作用。
In this paper, an interference-aware path planning scheme for a network of cellular-connected unmanned aerial vehicles (UAVs) is proposed. In particular, each UAV aims at achieving a tradeoff between maximizing energy efficiency and minimizing both wireless latency and the interference caused on the ground network along its path. The problem is cast as a dynamic game among UAVs. To solve this game, a deep reinforcement learning algorithm, based on echo state network (ESN) cells, is proposed. The introduced deep ESN architecture is trained to allow each UAV to map each observation of the network state to an action, with the goal of minimizing a sequence of time-dependent utility functions. Each UAV uses the ESN to learn its optimal path, transmission power, and cell association vector at different locations along its path. The proposed algorithm is shown to reach a subgame perfect Nash equilibrium upon convergence. Moreover, an upper bound and a lower bound for the altitude of the UAVs are derived thus reducing the computational complexity of the proposed algorithm. The simulation results show that the proposed scheme achieves better wireless latency per UAV and rate per ground user (UE) while requiring a number of steps that are comparable to a heuristic baseline that considers moving via the shortest distance toward the corresponding destinations. The results also show that the optimal altitude of the UAVs varies based on the ground network density and the UE data rate requirements and plays a vital role in minimizing the interference level on the ground UEs as well as the wireless transmission delay of the UAV.