CPS: Medium: Collaborative Research: Provably Safe and Robust Multi-Agent Reinforcement Learning with Applications in Urban Air Mobility

这个网络物理系统（CPS）项目旨在设计可扩展的多智能体规划和控制的理论和算法，以支持高吞吐量，不确定和动态环境中的安全关键自主eVTOL飞机。城市空中交通（UAM）是一种新兴的空中运输模式，其中电动垂直起降（eVTOL）飞机将在城市地区内安全有效地运输乘客和货物。来自白宫、国家工程院和美国国会的指导鼓励UAM的基础研究，以保持美国在该领域的全球领导地位。UAM的成功将取决于安全和强大的多智能体自主性，以将操作扩展到高吞吐量的城市空中交通。开发了基于学习的技术，如深度强化学习和多智能体强化学习，以支持这些垂直起降车辆的规划和控制。然而，在多智能体自主UAM应用中，为这些基于学习的神经网络在环模型提供理论上的安全性和鲁棒性保证是一个重大挑战。在这个项目中，研究人员将与政府和行业合作伙伴合作，开展基于用例的基础研究，重点是促进人工智能的安全性和可靠性，机器学习和不同背景学生的自主性。本项目的技术目标包括（1）单Agent强化学习的安全性和鲁棒性：为了解决“安全关键”的UAM挑战，PI计划单代理强化学习的最小-最大优化，以正式建立足够的安全裕度，约束强化学习将安全性公式化为状态和动作空间中的物理约束，以及一种新的谨慎强化学习方法，该方法使用变策略梯度来规划具有最小分布风险的最安全飞行器轨迹;（2）多智能体强化学习的安全性和鲁棒性：为了解决“异构代理和可伸缩性”的挑战，一种新的联合强化学习框架，其中中央代理与分散的安全代理协调以提高流量吞吐量，（3）从模拟到真实的世界的安全性和鲁棒性：为了解决“高维和环境不确定性”的挑战，研究人员将重点关注代理在分布转移和从模拟到真实的世界的快速适应下的策略鲁棒性。具体而言，计划进行以价值为目标的模型学习，以整合飞机和环境物理学等领域知识，并在RL模型在线部署用于飞行测试或执行后建立安全适应机制。该奖项反映了NSF的法定使命，通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This Cyber-Physical Systems (CPS) project aims at designing theories and algorithms for scalable multi-agent planning and control to support safety-critical autonomous eVTOL aircraft in high-throughput, uncertain and dynamic environments. Urban Air Mobility (UAM) is an emerging air transportation mode in which electrical vertical take-off and landing (eVTOL) aircraft will safely and efficiently transport passengers and cargo within urban areas. Guidance from the White House, the National Academy of Engineering, and the US Congress has encouraged fundamental research in UAM to maintain the US global leadership in this field. The success of UAM will depend on the safe and robust multi-agent autonomy to scale up the operations to high-throughput urban air traffic. Learning-based techniques such as deep reinforcement learning and multi-agent reinforcement learning are developed to support planning and control for these eVTOL vehicles. However, there is a major challenge to provide theoretical safety and robustness guarantees for these learning-based neural network in-the-loop models in multi-agent autonomous UAM applications. In this project, the researchers will collaborate with committed government and industry partners on the use-case-inspired fundamental research, with a focus on promoting safety and reliability of AI, machine learning and autonomy in students with diverse backgrounds. The technical objectives of this project include (1) Safety and Robustness of Single-Agent Reinforcement Learning: in order to address the “safety critical” UAM challenge, the PIs plan the min-max optimization for single agent reinforcement learning to formally build sufficient safety margin, constrained reinforcement learning to formulate safety as physical constraints in state and action spaces, and the novel cautious reinforcement learning that uses variational policy gradient to plan the safest aircraft trajectory with minimum distributional risk; (2) Safety and Robustness of Multi-Agent Reinforcement Learning: in order to address the “heterogeneous agents and scalability” challenge, a novel federated reinforcement learning framework where a central agent coordinates with decentralized safe agents to improve traffic throughput while guaranteeing safety, and a scaling mechanism to accommodate a varying number of decentralized aircraft; (3) Safety and Robustness from Simulations to the Real World: in order to address the “high-dimensionality and environment uncertainty” challenge, the researchers will focus on the agents’ policy robustness under distribution shift and fast adaptation from simulation to the real world. Specifically, value-targeted model learning to incorporate domain knowledge such as the aircraft and environment physics, and a safe adaptation mechanism after the RL model is deployed online for flight testing or execution is planned.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.