NSF-AoF: CNS Core: Small: Reinforcement Learning for Real-time Wireless Scheduling and Edge Caching: Theory and Algorithm Design

近年来，无线网络系统中的实时应用出现了巨大的增长，例如联网汽车和多用户增强现实（AR）。无线边缘缓存是另一个需要高带宽的新兴应用程序，其中最佳缓存决策将取决于缓存内容和动态用户需求配置文件。为了满足爆炸性需求，5G及以上（B5G）技术有望提供增强型移动宽带（eMBB）和超可靠低延迟通信（URLLC）服务。在无线网络中，满足URLLC要求是非常具有挑战性的，需要对当前的无线系统设计进行大规模修改。实时流量的截止时间感知无线调度一直是一个长期存在的开放性问题。该合作项目为应对这些挑战带来了范式转变，从而为各种物联网应用（包括B5G、自动驾驶、增强现实、智能健康和智慧城市）在超低延迟和高带宽方面的QoS保证提供了新的思路，使美国和芬兰都受益。拟议的研究还将通过课程开发、研究经验和外展，与pi机构的研究生、本科生和K-12学生的教育活动相结合。本项目利用离线强化学习（RL）的最新进展，研究B5G中的两个重要问题，即1)截止日期感知无线调度，以保证低延迟；2)边缘缓存，以实现高带宽内容交付。在推力1中，物理辅助离线RL将被设计用于训练截止日期感知调度策略。具体来说，Actor- critic （A-C）方法将用于调度策略的离线训练，包括两个阶段：1)通过行为克隆初始化Actor结构，2)通过物理辅助的A-C方法改进策略。以良好的基于模型的调度算法作为初始参与者结构，利用a - c方法可以产生更好的调度策略，这得益于其策略改进的性质。此外，将设计创新算法来解决A-C方法的突出问题，即过高估计偏差和高方差，并将Meta-RL用于适应非平稳网络动态中的分布移动。Thrust 2侧重于无线边缘缓存，这是一种利用网络边缘和用户设备的存储容量来缓解长距离高带宽通信需求的应用程序。联合通信和缓存优化的组合特性，以及系统动力学的不确定性，要求机器学习算法的非平凡设计。pi将利用RL彻底研究无线边缘缓存。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Recent years have witnessed a tremendous growth in real-time applications in wirelessly networked systems, such as connected cars and multi-user augmented reality (AR). Wireless edge caching is another emerging application requiring high bandwidth, where optimal caching decisions would depend on the cache contents and dynamic user demand profiles. To meet the explosive demand, 5G and Beyond (B5G) technology promises to offer enhanced mobile broadband (eMBB) and ultra-reliable low-latency communications (URLLC) services. Meeting URLLC requirements is very challenging in wireless networks, and requires massive modifications to the current wireless system design. Deadline-aware wireless scheduling of real-time traffic has been a long-standing open problem. This collaborative project makes a paradigm shift to tackle these challenges thus spurring a new line of thinking for QoS guarantee in terms of ultra-low latency and high bandwidth in a variety of IoT applications, including B5G, autonomous driving, augmented reality, smart health and smart city, benefiting both the US and Finland. The proposed research will also be integrated with education activities at the PIs' institutions for graduate, undergraduate, and K-12 students via curriculum development, research experiences, and outreach. This project leverages recent advances on offline reinforcement learning (RL) to study two important problems in B5G, namely 1) deadline-aware wireless scheduling to guarantee low latency and 2) edge caching to achieve high bandwidth content delivery. In Thrust 1, physics-aided offline RL will be devised to train deadline-aware scheduling policies. Specifically, the Actor-Critic (A-C) method will be used for offline training of scheduling policies, consisting of two phases: 1) initialization of Actor structure via behavioral cloning and 2) policy improvement via the physics-aided A-C method. With a good model-based scheduling algorithm as the initial actor structure, the A-C method can be leveraged to yield a better scheduling policy, thanks to its nature of policy improvement. Further, innovative algorithms will be devised to address the outstanding problems in the A-C method, namely overestimation bias and high variance, and Meta-RL will be used for adaptation to distribution shift in nonstationary network dynamics. Thrust 2 focuses on wireless edge caching, an application where the storage capacities at both the network edge and user devices are harnessed to alleviate the need of high-bandwidth communications over long distances. The combinatorial nature of joint communication and caching optimization herein, with the uncertainties of system dynamics, calls for non-trivial design of machine learning algorithms. The PIs will leverage RL to investigate wireless edge caching thoroughly.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.