CAREER: Foundations of Scalable and Resilient Distributed Real-Time Decision Making in Open Multi-Agent Systems

人工智能和机器学习的进步为使用自主多智能体系统解决重要的社会和经济问题提供了机会，例如多个机器人在野火监测，搜索和救援，制造等方面的应用。强化学习是一种数据驱动的控制方法，它使智能体能够通过直接与环境交互来自主学习所需的任务，已经成为这种实时决策的主要框架之一。虽然强化学习提供了一个强大而灵活的框架，但它在可扩展性和弹性方面面临着根本性的挑战。具体而言，现有方法需要大量的数据和计算能力，并且在存在各种类型的错误和对手的情况下可能不稳定。这些挑战是强化学习广泛应用于现实世界问题的主要障碍。这个CAREER项目将为开放多智能体系统中的实时自主合作开发可扩展和弹性分布式强化学习的新基础。总体目标是设计新的学习和控制方法，使代理在开放系统中有效地进行交互，优雅地适应时变环境，并对意外故障和对手具有弹性。该项目还将通过将研究成果与严格的教育和推广活动、课程开发、学生培训和公共伙伴关系相结合，为教育和劳动力发展做出贡献。该项目的中心思想是建立非单调系统的双时间尺度随机逼近的新基础。关键的方法是利用外推技术在优化和奇异摄动理论的控制，以解决非单调设置下的随机逼近的不稳定性问题。将研究新的理论原则，以表征所提出的方法的有限时间复杂度。通过利用这些新的两个时间尺度的随机近似的结果，该项目将推进开放的多智能体系统中的分布式学习和控制的几个基本方面。重点是开发新的可扩展和弹性的分布式多时间尺度强化学习方法，允许代理在各种实际考虑因素下实时有效地合作，包括时变代理数量，意外故障，通信约束和对手。在本项目实施过程中，将通过一系列多机器人导航的模拟和现场实验，对拟议的研究活动进行系统评估。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Advances in artificial intelligence and machine learning provide the opportunity for using autonomous multi-agent systems to solve important social and economic problems, such as the application of multiple robots in wildfire monitoring, search-and-rescue, manufacturing, etc. In these systems, agents autonomously cooperate to make decisions in real-time to perform complex tasks. Reinforcement learning, a data-driven control method that enables agents to autonomously learn desired tasks by interacting directly with the environment, has emerged as one of the predominant frameworks for this kind of real-time decision making. While reinforcement learning provides a powerful and flexible framework, it suffers fundamental challenges in its scalability and resilience. Specifically, existing methods require a vast amount of data and computational power and can be unstable in the presence of various types of errors and adversaries. These challenges are the main barriers to the wide applicability of reinforcement learning for real-world problems. This CAREER project will develop new foundations of scalable and resilient distributed reinforcement learning for real-time autonomous cooperation in open multi-agent systems. The overarching goal is to design new learning and control methods that enable agents to interact effectively in open systems, adapt gracefully in time-varying environments, and be resilient to unexpected failures and adversaries. The project will also contribute to education and workforce development by integrating the research findings with rigorous educational and outreach activities, course development, student training, and public partnerships.The central idea of this project is to establish new fundamentals of two-time-scale stochastic approximation for non-monotone systems. The key approach is to leverage extrapolation techniques in optimization and singular perturbation theories in control to address the instability issues of stochastic approximation under non-monotone settings. New theoretical principles will be studied to characterize the finite-time complexity of the proposed methods. By leveraging these new results of two-time-scale stochastic approximation, this project will advance several foundational aspects of distributed learning and control in open multi-agent systems. The focus is to develop new scalable and resilient distributed multi-time-scale reinforcement learning methods that allow agents to cooperate efficiently in real-time under diverse practical considerations, including time-varying numbers of agents, unexpected failures, communication constraints, and adversaries. During the course of this project, the proposed research activities will be evaluated systematically through a series of simulations and field experiments of multi-robot navigation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.