CNS Core: Small: Online Safe Reinforcement Learning for Wireless Resource Allocation

下一代无线网络正在进行设计，以满足复杂的应用需求组合，从传统的移动宽带(例如，网页浏览、视频流)到具有异类更严格的可靠性-延迟要求的新兴应用(例如，增强现实、自动驾驶汽车、工业自动化、机器人)。支持这些要求的能力推动了交付新业务模式和新收入流的潜力，这将使新技术的部署成为可能。因此，无线调度和资源分配将在支持此类网络的技术方面占据中心地位。同时，使用深度网络的强化学习(RL)已经成为设计优化复杂系统(包括无线系统)性能的策略的强大框架；然而，这些通常没有任何正式的保证。这项研究的中心论点是，基于RL的资源分配策略不太可能被接受和/或部署，因此，开发确保安全保证的方法是基于RL的资源分配策略不太可能被接受和/或部署。这项研究推进了安全强化学习的最新发展，特别是在无线系统中的应用，但也有望通过在教育、创新、实现多样性、参与社区和行业以及向更广泛的公众传播结果方面的有计划的努力，造福于其他应用领域和更广泛的社会。本研究集中在开发和分析安全强化学习(Safe-RL)框架，该框架在短时间尺度上优化奖励，并为最先进的无线调度算法提供理论上强大的长期吞吐量优化保证。关键的潜在观察是，今天的许多调度算法从李亚普诺夫分析中获得其性能保证。该项目利用了护栏的创新概念--对SAFE-RL的状态依赖动作的约束--保证无线系统的李亚普诺夫演化保持在经典算法的有界扰动范围内。这一保证进而确保Safe-RL具有最先进的调度器的安全/稳定属性，同时利用RL实现复杂的性能权衡。这项研究包括三个相互关联的推动力。该项目的核心是开发SAFE-RL的基础和表示法，以及安全保证的理论基础和无线系统网络抽象的有效学习的新类别。推力2侧重于将安全-RL理论应用于无线资源分配，包括解决与实时和宽带流量的联合调度相关的挑战、学习和利用流量模式，以及探索作为系统异常或需要重新优化的指示的策略遇到护栏的程度。该奖项反映了美国国家科学基金会的法定使命，通过使用基金会的智力优势和更广泛的影响审查标准进行评估，认为值得给予支持。

Next generation wireless networks are being engineered to meet a complex mix of application requirements, from traditional mobile broadband (e.g., web browsing, video streaming) to new emerging applications (e.g., augmented reality, self-driving cars, industrial automation, robotics) with heterogeneous much more stringent reliability-latency requirements. The ability to support these requirements drives the potential to deliver the new business models and new revenue streams that would enable deployments of the new technology. Thus, wireless scheduling and resource allocation will take center stage in terms of enabling technologies for such networks. Meanwhile reinforcement learning (RL) using deep networks has emerged as a powerful framework to devise polices that optimize complex systems' performance (including wireless systems); however, these usually do not come with any formal guarantees. The central thesis of this research is that RL-based resource allocation policies without operational guarantees, e.g., throughput-optimality/stability, are unlikely to be accepted and/or deployed, thus a key requirement to make these techniques usable, is to develop approaches which ensure safety guarantees. The research advances the state-of-the-art in safe reinforcement learning, with specific applications to wireless systems, but also is expected to benefit other application domains as well as society more broadly, through planned efforts in education, innovation, achieving diversity, engaging the community and industry, and disseminating results to a wider public.This research centers on the development and analysis of a safe reinforcement learning (Safe-RL) framework, which optimizes rewards over short-time scales, and also provides theoretically strong long-term throughput-optimality guarantees for state-of-art wireless scheduling algorithms. The key underlying observation is that many of today's scheduling algorithms derive their performance guarantees from Lyapunov analysis. The project leverages the innovative concept of guardrails -- constraints on the state-dependent actions of Safe-RL -- that guarantee that the wireless system's Lyapunov evolution stay within a bounded perturbation of classical algorithms. This guarantee, in turn, ensures that Safe-RL has safety/stability properties of state-of-the-art schedulers, while leveraging RL to realize complex performance tradeoffs. The research consists of three inter-related thrusts. Thrust 1 develops the foundations and representations for safe-RL at the core of this project, along with a theoretical basis for safety guarantees and new classes of efficient learning for wireless system network abstractions. Thrust 2 focuses on the application of safe-RL theory to wireless resource allocation, including addressing challenges associated with joint scheduling of real-time and broadband traffic, learning and exploiting traffic patterns, and an exploration of the degree to which a policy hits guardrails as an indication of system anomalies or need for re-optimization. Thrust 3 centers on the challenging but necessary task of validating the safe-RL framework leveraging an industrial strength multi-cell simulator.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.