CPS: Medium: Collaborative Research: Provably Safe and Robust Multi-Agent Reinforcement Learning with Applications in Urban Air Mobility

这个网络物理系统(CPS)项目旨在设计可扩展的多智能体规划和控制的理论和算法，以支持高吞吐量、不确定和动态环境中的安全关键自主eVTOL飞机。城市空中机动性(UAM)是一种新兴的航空运输方式，电动垂直起降(EVTOL)飞机在城市区域内安全、高效地运送乘客和货物。白宫、美国国家工程院和美国国会的指导鼓励了UAM的基础研究，以保持美国在该领域的全球领先地位。UAM的成功将取决于安全和强大的多代理自治，以将运营扩大到高吞吐量的城市空中交通。基于学习的技术，如深度强化学习和多智能体强化学习，被开发来支持这些eVTOL车辆的规划和控制。然而，在多智能体自主UAM应用中，为这些基于学习的神经网络在环模型提供理论上的安全性和健壮性保证是一个重大挑战。在这个项目中，研究人员将与承诺的政府和行业合作伙伴合作，开展以用例为灵感的基础研究，重点是促进人工智能、机器学习和不同背景学生的自主性的安全和可靠性。本项目的技术目标包括：(1)单智能体强化学习的安全性和稳健性：为了解决“安全关键”的UAM挑战，PI计划对单智能体强化学习进行最小-最大优化以形式化地建立足够的安全裕度，约束强化学习将安全性表达为状态和动作空间中的物理约束，以及新的谨慎强化学习，它使用变化的策略梯度来规划最安全的飞机轨迹并使分布风险最小；(2)多智能体强化学习的安全性和健壮性：为了解决“异质智能体和可伸缩性”的挑战，提出了一种新型的联邦强化学习框架，其中中央智能体与分散的安全智能体进行协调，以提高交通吞吐量，同时保证安全，并建立了一个缩放机制，以适应不同数量的分散式飞机；(3)从模拟到真实世界的安全性和健壮性：为了应对“高维和环境不确定性”的挑战，研究人员将重点关注分布转移下的智能体的策略健壮性以及从模拟到真实世界的快速适应。具体地说，计划进行以价值为目标的模型学习，以纳入飞机和环境物理等领域知识，并在RL模型在线部署进行飞行测试或执行后建立安全适应机制。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This Cyber-Physical Systems (CPS) project aims at designing theories and algorithms for scalable multi-agent planning and control to support safety-critical autonomous eVTOL aircraft in high-throughput, uncertain and dynamic environments. Urban Air Mobility (UAM) is an emerging air transportation mode in which electrical vertical take-off and landing (eVTOL) aircraft will safely and efficiently transport passengers and cargo within urban areas. Guidance from the White House, the National Academy of Engineering, and the US Congress has encouraged fundamental research in UAM to maintain the US global leadership in this field. The success of UAM will depend on the safe and robust multi-agent autonomy to scale up the operations to high-throughput urban air traffic. Learning-based techniques such as deep reinforcement learning and multi-agent reinforcement learning are developed to support planning and control for these eVTOL vehicles. However, there is a major challenge to provide theoretical safety and robustness guarantees for these learning-based neural network in-the-loop models in multi-agent autonomous UAM applications. In this project, the researchers will collaborate with committed government and industry partners on the use-case-inspired fundamental research, with a focus on promoting safety and reliability of AI, machine learning and autonomy in students with diverse backgrounds. The technical objectives of this project include (1) Safety and Robustness of Single-Agent Reinforcement Learning: in order to address the “safety critical” UAM challenge, the PIs plan the min-max optimization for single agent reinforcement learning to formally build sufficient safety margin, constrained reinforcement learning to formulate safety as physical constraints in state and action spaces, and the novel cautious reinforcement learning that uses variational policy gradient to plan the safest aircraft trajectory with minimum distributional risk; (2) Safety and Robustness of Multi-Agent Reinforcement Learning: in order to address the “heterogeneous agents and scalability” challenge, a novel federated reinforcement learning framework where a central agent coordinates with decentralized safe agents to improve traffic throughput while guaranteeing safety, and a scaling mechanism to accommodate a varying number of decentralized aircraft; (3) Safety and Robustness from Simulations to the Real World: in order to address the “high-dimensionality and environment uncertainty” challenge, the researchers will focus on the agents’ policy robustness under distribution shift and fast adaptation from simulation to the real world. Specifically, value-targeted model learning to incorporate domain knowledge such as the aircraft and environment physics, and a safe adaptation mechanism after the RL model is deployed online for flight testing or execution is planned.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.