CAREER: Foundations of Scalable and Resilient Distributed Real-Time Decision Making in Open Multi-Agent Systems

人工智能和机器学习的发展为使用自主多智能体系统来解决重要的社会和经济问题提供了机会，如多机器人在野火监测、搜救、制造等方面的应用。在这些系统中，智能体自主协作，实时做出决策，执行复杂的任务。强化学习是一种数据驱动的控制方法，它使智能体能够通过与环境直接交互来自主学习期望的任务，已经成为这种实时决策的主要框架之一。虽然强化学习提供了一个强大而灵活的框架，但它在可扩展性和弹性方面面临着根本性的挑战。具体地说，现有的方法需要大量的数据和计算能力，并且在存在各种类型的错误和对手时可能不稳定。这些挑战是强化学习广泛应用于现实世界问题的主要障碍。该项目将为开放多智能体系统中的实时自主协作开发可扩展和弹性的分布式强化学习的新基础。总体目标是设计新的学习和控制方法，使代理能够在开放系统中有效地交互，在时变的环境中优雅地适应，并对意外的故障和对手具有弹性。该项目还将通过将研究成果与严格的教育和外展活动、课程开发、学生培训和公共合作伙伴关系相结合，为教育和劳动力发展做出贡献。该项目的中心思想是为非单调系统建立新的两时间尺度随机逼近的基本原理。关键的方法是利用最优化中的外推技术和控制中的奇异摄动理论来解决非单调设置下随机逼近的不稳定性问题。将研究新的理论原理来表征所提出方法的有限时间复杂性。通过利用这些双时间尺度随机逼近的新结果，该项目将推进开放多代理系统中分布式学习和控制的几个基本方面。其重点是开发新的可扩展和弹性的分布式多时间尺度强化学习方法，使代理能够在不同的实际考虑因素下实时有效地合作，包括时变的代理数量、意外故障、通信约束和对手。在这个项目的过程中，建议的研究活动将通过一系列多机器人导航的模拟和现场实验进行系统的评估。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Advances in artificial intelligence and machine learning provide the opportunity for using autonomous multi-agent systems to solve important social and economic problems, such as the application of multiple robots in wildfire monitoring, search-and-rescue, manufacturing, etc. In these systems, agents autonomously cooperate to make decisions in real-time to perform complex tasks. Reinforcement learning, a data-driven control method that enables agents to autonomously learn desired tasks by interacting directly with the environment, has emerged as one of the predominant frameworks for this kind of real-time decision making. While reinforcement learning provides a powerful and flexible framework, it suffers fundamental challenges in its scalability and resilience. Specifically, existing methods require a vast amount of data and computational power and can be unstable in the presence of various types of errors and adversaries. These challenges are the main barriers to the wide applicability of reinforcement learning for real-world problems. This CAREER project will develop new foundations of scalable and resilient distributed reinforcement learning for real-time autonomous cooperation in open multi-agent systems. The overarching goal is to design new learning and control methods that enable agents to interact effectively in open systems, adapt gracefully in time-varying environments, and be resilient to unexpected failures and adversaries. The project will also contribute to education and workforce development by integrating the research findings with rigorous educational and outreach activities, course development, student training, and public partnerships.The central idea of this project is to establish new fundamentals of two-time-scale stochastic approximation for non-monotone systems. The key approach is to leverage extrapolation techniques in optimization and singular perturbation theories in control to address the instability issues of stochastic approximation under non-monotone settings. New theoretical principles will be studied to characterize the finite-time complexity of the proposed methods. By leveraging these new results of two-time-scale stochastic approximation, this project will advance several foundational aspects of distributed learning and control in open multi-agent systems. The focus is to develop new scalable and resilient distributed multi-time-scale reinforcement learning methods that allow agents to cooperate efficiently in real-time under diverse practical considerations, including time-varying numbers of agents, unexpected failures, communication constraints, and adversaries. During the course of this project, the proposed research activities will be evaluated systematically through a series of simulations and field experiments of multi-robot navigation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.