CNS Core: Small: Online Safe Reinforcement Learning for Wireless Resource Allocation

下一代无线网络的设计是为了满足复杂的应用需求组合，从传统的移动宽带（例如，网页浏览、视频流）到新兴的应用（例如，增强现实、自动驾驶汽车、工业自动化、机器人），这些应用具有更严格的可靠性和延迟要求。支持这些需求的能力推动了交付新的业务模型和新的收入流的潜力，这些将支持新技术的部署。因此，无线调度和资源分配将成为这类网络的核心技术。与此同时，使用深度网络的强化学习（RL）已经成为一个强大的框架，用于设计优化复杂系统（包括无线系统）性能的策略；然而，这些通常没有任何正式的保证。本研究的中心论点是，没有操作保证（例如，吞吐量最优性/稳定性）的基于rl的资源分配策略不太可能被接受和/或部署，因此，使这些技术可用的关键要求是开发确保安全保证的方法。该研究推进了安全强化学习的最新技术，具体应用于无线系统，但也有望通过在教育、创新、实现多样性、参与社区和行业以及向更广泛的公众传播结果方面的计划努力，更广泛地造福于其他应用领域和社会。本研究以开发和分析安全强化学习（safe - rl）框架为中心，该框架在短时间尺度上优化奖励，并为最先进的无线调度算法提供理论上强大的长期吞吐量最优性保证。关键的潜在观察是，今天的许多调度算法从Lyapunov分析中获得性能保证。该项目利用了护栏的创新概念——对Safe-RL状态依赖行为的约束——确保无线系统的Lyapunov进化保持在经典算法的有界扰动范围内。这种保证反过来又确保了Safe-RL具有最先进调度器的安全/稳定特性，同时利用RL实现复杂的性能权衡。这项研究包括三个相互关联的重点。推力1作为该项目的核心，开发了安全强化学习的基础和表示，以及安全保证的理论基础和无线系统网络抽象的高效学习的新类别。推力2侧重于安全rl理论在无线资源分配中的应用，包括解决与实时和宽带流量联合调度相关的挑战，学习和利用流量模式，以及探索策略撞击护栏的程度，作为系统异常或需要重新优化的指示。推力3的重点是利用工业强度的多单元模拟器验证安全rl框架，这是一项具有挑战性但必要的任务。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Next generation wireless networks are being engineered to meet a complex mix of application requirements, from traditional mobile broadband (e.g., web browsing, video streaming) to new emerging applications (e.g., augmented reality, self-driving cars, industrial automation, robotics) with heterogeneous much more stringent reliability-latency requirements. The ability to support these requirements drives the potential to deliver the new business models and new revenue streams that would enable deployments of the new technology. Thus, wireless scheduling and resource allocation will take center stage in terms of enabling technologies for such networks. Meanwhile reinforcement learning (RL) using deep networks has emerged as a powerful framework to devise polices that optimize complex systems' performance (including wireless systems); however, these usually do not come with any formal guarantees. The central thesis of this research is that RL-based resource allocation policies without operational guarantees, e.g., throughput-optimality/stability, are unlikely to be accepted and/or deployed, thus a key requirement to make these techniques usable, is to develop approaches which ensure safety guarantees. The research advances the state-of-the-art in safe reinforcement learning, with specific applications to wireless systems, but also is expected to benefit other application domains as well as society more broadly, through planned efforts in education, innovation, achieving diversity, engaging the community and industry, and disseminating results to a wider public.This research centers on the development and analysis of a safe reinforcement learning (Safe-RL) framework, which optimizes rewards over short-time scales, and also provides theoretically strong long-term throughput-optimality guarantees for state-of-art wireless scheduling algorithms. The key underlying observation is that many of today's scheduling algorithms derive their performance guarantees from Lyapunov analysis. The project leverages the innovative concept of guardrails -- constraints on the state-dependent actions of Safe-RL -- that guarantee that the wireless system's Lyapunov evolution stay within a bounded perturbation of classical algorithms. This guarantee, in turn, ensures that Safe-RL has safety/stability properties of state-of-the-art schedulers, while leveraging RL to realize complex performance tradeoffs. The research consists of three inter-related thrusts. Thrust 1 develops the foundations and representations for safe-RL at the core of this project, along with a theoretical basis for safety guarantees and new classes of efficient learning for wireless system network abstractions. Thrust 2 focuses on the application of safe-RL theory to wireless resource allocation, including addressing challenges associated with joint scheduling of real-time and broadband traffic, learning and exploiting traffic patterns, and an exploration of the degree to which a policy hits guardrails as an indication of system anomalies or need for re-optimization. Thrust 3 centers on the challenging but necessary task of validating the safe-RL framework leveraging an industrial strength multi-cell simulator.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.