CAREER: Structure Exploiting Multi-Agent Reinforcement Learning for Large Scale Networked Systems: Locality and Beyond

该项目旨在开发一套多智能体强化学习（MARL）算法，用于控制网络系统，这些系统无处不在，在推进我们的现代社会中发挥着不可或缺的作用。例子跨越了广泛的范围，包括电力/能源网，运输系统，网络机器人等，这些系统的控制和操作长期以来一直是一个巨大的挑战，由于系统中的不同组件之间的复杂的相互依赖关系，越来越多的连接和相互作用的代理，以及越来越多的环境不确定性。 该项目将为网络系统的控制和操作方式带来变革性的变化，通过使用MARL设计新的策略来实现，在复杂和不确定的环境下，效率和可靠性将有数量级的提高。 知识价值集中在MARL的独特结构开发方法。大多数现有的MARL方法采用底层系统的黑盒视图，而不利用底层结构，并且已知具有可扩展性和稳定性/安全性问题。相比之下，现实世界的系统具有丰富的结构属性，这些属性很容易利用，例如电网中的连通性拓扑。鉴于上述结构特性，该项目利用底层结构来设计可扩展的，稳定的，安全的MARL的大规模网络系统。该框架不仅在理论上合理，而且具有广泛的适用性。为了展示该方法的广泛适用性，本项目将在多服务器系统中的电力系统和负载平衡上进行演示。 更广泛的影响引入了一个独特的结构开发视角，即机器学习应该与特定工程应用的结构特性相结合。该项目包括扩大这一观点影响的计划。除了研究本身，该项目还包括协同教育，多样性和扩大参与计划：（a）在机器学习和工程网络系统的交叉点开发新课程;（B）通过CMU Gelfand外展计划开发K12课程和外展活动;以及（c）该奖项反映了国家科学基金会的法定使命，并被认为是值得通过使用基金会的智力价值和更广泛的影响进行评估的支持审查标准。

This project aims to develop a suite of Multi-Agent Reinforcement Learning (MARL) algorithms for the control of networked systems, which are ubiquitous and play an indispensable role in advancing our modern society. Examples cut across a broad spectrum, including power/energy grid, transportation systems, networked robots, etc. The control and operation of such systems have long been a tremendous challenge, due to the complex interdependence between different components in the system, the increasing number of connected and interacting agents, and the increasing environmental uncertainties. The project will bring transformative change to how networked systems are controlled and operated, achieved by using MARL to design novel policies with order-of-magnitude improvement in efficiency and reliability under complex and uncertain environments. The intellectual merit focuses on a unique structure-exploiting approach for MARL. Most of the existing MARL approaches take a black-box view of the underlying system without utilizing the underlying structure and are known to have scalability and stability/safety issues. In contrast, real-world systems have rich structural properties that are readily available to exploit, e.g. the connectivity topology in the power grid. In light of the above structural properties, the project exploits the underlying structure to design scalable, stable, and safe MARL for large-scale networked systems. The proposed framework is not only theoretically sound but also widely applicable. To show the wide applicability of the approach, this project will demonstrate it on power systems and load balancing in multiserver systems. The broader impacts introduce a unique structure-exploiting perspective, that is machine learning should be integrated with the structural property of specific engineering applications. The project includes plans to broaden the impact of this perspective. Beyond the research itself, the project includes a synergistic education, diversity, and a broadening participation plan: (a) development of a new course at the intersection of machine learning and engineering networked systems; (b) K12 curriculum development and outreach activities through the CMU Gelfand Outreach program; and (c) undergraduates research through the SURF program.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.