ERI: Distributed Learning in Regulation of UAV Communication Networks with Dynamic UAV Lineup

该奖项全部或部分由《2021年美国救援计划法案》（公法117-2）资助。无人驾驶飞行器在下一代无线通信领域展示了令人印象深刻的潜力。与地面蜂窝基站相比，配备无线收发器的无人机可以作为移动基站，具有灵活的3D移动性、更好的无线连接性和更高的视距链路机会、更低的部署成本和几乎不需要基础设施的网络建设等特点，可以提供高度按需的服务。尽管前景广阔，但基于无人机的通信网络仍然面临着根本性的监管挑战：1)由于无人机的高度移动性，动态变化的网络拓扑结构可能使集中控制单元无法收集完整的网络信息并进行集体决策；Ii)无人机操作的环境可能是动态变化的或未经探索的，没有对环境建模的先验知识，使得传统的优化或基于规则的方法很难适用；Iii)当服务无人机的阵容发生动态变化时，如何对无人机网络进行最优调节的研究仍处于萌芽阶段，但这将是现实实现中常见的事件。为此，本提案旨在破解上述问题的难题，并为分布式网络监管解决方案制定一个有效的框架，该框架不受环境模型的影响，并能很好地适应动态无人机阵容。该研究成果有望为动态网络设置下的分布式、可扩展和人工智能驱动的空中接入通信网络管理提供有价值的启发和基准。这些网络将成为超5g移动通信的更大范围天空地一体化网络的关键组成部分。该项目的成功将有助于美国在未来一代移动通信领域的全球领导地位和竞争力，并通过更综合和按需通信基础设施提高国家通信福利。在本项目中，将应用多智能体强化学习来建立一个分布式、无模型的网络监管框架。该框架具有在复杂时变环境下进行序列决策的强大能力。在开发的框架下，该项目旨在研究无人机通信网络如何以分布式协调的方式响应处理并进一步主动控制无人机阵容的动态变化。具体来说，响应策略将首先设计用于一般无人机通信网络。该策略将共同优化无人机编队动态变化时的无线电资源管理和轨迹设计。研究了不同层次的无人机间信息交换的学习算法设计。采用异步并行计算的结构将促进学习探索。该网络将利用可编程无人机产品和简单而有效的通信协议进行原型设计。为了更进一步，将为太阳能自主可持续无人机通信网络导出主动控制策略，该网络通过预先制定无人机的太阳能充电计划，主动控制无人机的退出和加入。该策略将考虑动态用户空间和流量分布，通过结合傅里叶分析、长短期记忆和高斯过程回归进行分布预测，并实现边学习边预测，从而显著降低强化学习的复杂性。利用纳什Q学习和相关Q学习来处理无人机个体间的混合合作-竞争关系。采用问题分解技术可以缓解学习过程中对高维状态-动作空间的焦虑。该项目将填补控制策略设计缺失的空白，以响应式处理和主动控制无人机阵容变化，从而推进无人机通信网络的自主调节研究。此外，将博弈论引入到分布式框架中，使得个体无人机具有多样化的自主性，使研究更具现实性。原型计划将补充对现实世界实现中时间复杂性和通信开销/延迟的模拟评估。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Unmanned aerial vehicles, or drones, have been demonstrating impressive potentials in next generation wireless communications. Compared to the terrestrial cellular base stations, drones equipped with wireless transceivers can serve as mobile base stations, and stand out in providing highly on-demand services with flexible 3D mobility, better wireless connectivity with higher chance of Line-of-Sight links, and much lower deployment cost with almost infrastructure-free network construction. Promising as it is, drone based communication networks still face fundamental regulation challenges: i) as drones are highly mobile, the dynamically changing network topology may make it unpractical for a centralized control unit to collect complete network information and make collective decisions; ii) the environment in which drones operate may be dynamically changing or unexplored without a priori knowledge of the environment modeling, making the conventional optimization or rule-based methods hardly applicable; iii) the research is still embryonic on how to optimally regulate the drone network when the lineup of the serving drones dynamically change, which, however, will be a common event in realistic implementation. To this end, this proposal is aimed to crack the nut of the above identified issues, and develop an effective framework for distributed network regulation solutions that are environment-model-free and well adaptive to the dynamic drone lineup. The research outcomes are expected to provide valuable inspirations and benchmarking to the distributed, scalable, and artificial-intelligence powered management of aerial access communication networks under a dynamic network setup. Such networks will be embraced as a key component in the larger-scope Space-Air-Ground Integrated Networks for the beyond-5G mobile telecommunications. The success of the project will potentially contribute to the leadership and competence of the United States worldwide in future generation mobile telecommunications as well as elevating national communication welfare with more integrated and on-demand communication infrastructure.In this project, multi-agent reinforcement learning will be applied to establish a distributed and model-free network regulation framework. The framework will feature strong capability in making sequential decisions in complex time-varying environments. Under the developed framework, the project aims to investigate how the drone communication networks should responsively handle and further proactively control the dynamic change of the drone lineup in a distributed yet coordinated manner. Specifically, responsive strategies will be first designed for a general drone communication network. The strategies will jointly optimize the radio resource management and trajectory design for the drones when the drone lineup change dynamically. The learning algorithm design will be investigated with different levels of inter-drone information exchange. The learning exploration will be promoted by adopting the structure of asynchronous parallel computing. The network will be prototyped leveraging on programmable drone products and simple-yet-effective communication protocols. To move one step further, proactive control strategies will be derived for the solar-powered self-sustainable drone communication network, which proactively control the quit and join-in of the drones by pre-shaping their solar-charging plan. The strategies will consider dynamic user spatial and traffic distributions by combining Fourier analysis, Long-Short Term Memory and Gaussian process regression for distribution prediction, and enabling predicting while learning to significantly reduce the reinforcement learning complexity. The hybrid cooperative-compete relationship among individual drones will be handled by exploiting Nash Q learning and correlated Q learning. The anxiety on the high-dimension state-action space in learning will be relieved by adopting problem decomposition techniques. The proposed project will advance the research on autonomous regulation of drone communication networks by filling the gaps of missing control strategy design to responsively handle and proactively control the drone lineup change. In addition, the introduction of game theory into the distributed framework makes the research more realistic with diversified autonomy in individual drones. The prototyping plan will complement the simulation evaluation on the time complexity and communication overhead/latency in the real-world implementation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.