CAREER: Reinforcement Learning-Based Control of Heterogeneous Multi-Agent Systems in Structured Environments: Algorithms and Complexity

强化学习是一种在复杂环境下学习最优决策的流行框架，已有许多强化学习算法被用来改善单个智能体在正常环境下的决策。然而，现代大规模分布式学习应用程序通常涉及多个与复杂环境交互的异质代理，这使得最优决策从根本上更具挑战性。例如，在开阔区域导航多架无人机时，无人机需要适当地相互配合，并考虑到环境的不确定性。作为另一个例子，在分布式无线网络中，智能体(如基站或移动电话)之间的交互受到功率和带宽等异质约束。本项目旨在开发一个弹性RL框架来管理复杂环境中的异质多智能体系统，并系统地设计具有全面收敛和复杂性分析的高效多智能体RL算法。该项目将产生完全向公众开放的RL算法包。研究活动还将对本科生和研究生产生积极的教育影响。该项目开发的材料将被整合到机器学习和优化课程中，并将造福于电气和计算机工程、统计和计算机科学专业的跨学科学生。该项目将积极吸收代表性不足的学生参与，并将研究与STEM本科生和研究生的教育相结合。这个项目的主要目标是开发一个弹性的RL框架来管理涉及复杂和结构化环境中的异类代理的多代理系统，并系统地设计可扩展的、计算高效的RL算法来管理这样的系统，具有严格和全面的收敛和复杂性分析。这项拟议的研究包括三个主要推动力。首先，为了在各种类型的结构化环境(如同质性和不确定性)中管理具有异构性约束的协作代理，将利用环境模型结构来开发具有收敛和复杂性分析的完全分散的策略优化算法。其次，针对不确定环境下具有异质约束的竞争智能体的管理问题，提出了约束均衡和稳健均衡的新概念。研究了它们的基本结构和性质，并在此基础上提出了完全分散的原对偶型策略优化算法和具有收敛保证的健壮值基算法。最后，为了提高智能体策略在异质环境中的泛化能力，提出了一种新的辅助性RL框架，该框架可以在不共享数据的情况下，通过几轮信息交换来提高泛化能力。这些RL算法将用于学习用于无线网络中的干扰管理和电力网络中的能量控制的弹性和最佳控制策略。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Reinforcement learning (RL) is a popular framework for learning optimal decision-making in complex environments, and many RL algorithms have been developed to improve decision-making of a single agent in normal environments. However, modern large-scale distributed learning applications usually involve multiple heterogeneous agents that interact with complex environments, making the optimal decision-making fundamentally more challenging to learn. For example, when navigating multiple drones in an open area, the drones need to properly cooperative with each other and take the environment uncertainty into account. As another example, in distributed wireless networks, the interaction of the agents (e.g., base stations or mobile phones) are subject to heterogeneous constraints on power and bandwidth, etc. This project aims to develop a resilient RL framework for managing heterogeneous multi-agent systems in complex environments, and systematically design efficient multi-agent RL algorithms with comprehensive convergence and complexity analysis. The project will produce RL algorithm packages that are fully accessible to the public. The research activities will also generate positive educational impacts on undergraduate and graduate students. The materials developed by this project will be integrated into courses on machine learning and optimization, and will benefit interdisciplinary students majoring in electrical and computer engineering, statistics and computer science. The project will actively involve underrepresented students and integrate research with education for undergraduate and graduate students in STEM. It will also produce introductory materials for K-12 students to be used in engineering summer research programs.The overarching goal of this project is to develop a resilient RL framework for managing multi-agent systems that involve heterogeneous agents in complex and structured environments, and systematically design scalable and computation-efficient RL algorithms with rigorous and comprehensive convergence and complexity analysis for managing such systems. The proposed research includes three major thrusts. First, to manage cooperative agents with heterogeneous constraints in various types of structured environments (e.g., homogeneity and uncertainty), the environment model structure will be leveraged to develop fully decentralized policy optimization algorithms with convergence and complexity analysis. Second, to manage competitive agents with heterogeneous constraints in uncertain environment, new tractable notions of constrained and robust equilibrium will be proposed. Their fundamental structures and properties will be studied, based on which fully-decentralized primal-dual type policy optimization algorithms and robust value-based algorithms with convergence guarantees will be developed. Lastly, to improve the generalizability of agents’ policies across heterogeneous environments, a new assistive RL framework that can substantially enhance the generalizability using few rounds of information exchange without data sharing will be developed. These RL algorithms will be applied to learn resilient and optimal control policies for interference management in wireless networks and energy control in power networks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.