Collaborative Research: SLES: Safe Distributional-Reinforcement Learning-Enabled Systems: Theories, Algorithms, and Experiments

强化学习（RL）在自动化和机器人领域取得了成功，被广泛认为是下一代学习驱动系统最重要的技术之一。例如，6G网络系统、自动驾驶、数字医疗和智慧城市都是由RL实现的。然而，尽管在过去几十年里取得了重大进展，但在实践中应用强化学习的一个主要障碍是缺乏“安全”保证，如鲁棒性、对尾部风险的弹性、操作约束等。这是因为传统强化学习的目标只是最大化累积奖励。虽然在传统的强化学习算法中可以在奖励中添加惩罚来阻止不安全的行为，但许多安全约束（如机会约束）不能简单地视为惩罚。该项目开发了基于分布式强化学习（DRL）的安全强化学习系统的基础技术，该系统可以学习最优策略。在为安全学习系统开发DRL基础的同时，通过将该项目开发的新理论和算法纳入其研究生课程，将研究和教育相结合。所有团队成员都定期监督本科生和来自代表性不足群体的学生。该团队继续利用俄亥俄州立大学的女性之家和亚利桑那州立大学的女性科学与工程项目来提高女性学生和研究人员的广泛参与。该项目侧重于为支持drl的系统提供端到端安全的综合方法。端到端安全包括(i)策略安全：学习安全策略以避免灾难性结果的发生（对应于风险敏感RL）；（ii）探索安全——通过在探索/学习过程中避免危险行为安全地学习安全策略（对应于在线RL）；（iii）环境安全——学习一种对参数不确定性（环境变化）具有鲁棒性的策略。这个项目包括四个重点。推力1 （Foundation of constrained DRL）旨在建立风险敏感型约束DRL的理论基础，重点关注政策和环境安全。推力2 （Online constrained DRL）考虑安全的在线学习和决策，在学习安全的DRL策略时，重点关注勘探安全和环境安全。推力3（物理增强约束DRL）利用物理来增强端到端安全性。基础研究的这三个重点是相互依存的，但每个重点都关注安全强化学习系统的一个独特方面，并解决多个安全概念。第四次推力将通过高保真仿真和使用无人机的真实世界实验提供全面验证。这项研究得到了美国国家科学基金会和开放慈善机构的合作支持。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Reinforcement learning (RL), with its success in automation and robotics, has been widely viewed as one of the most important technologies for next-generation, learning-enabled systems. For example, 6G networking systems, autonomous driving, digital healthcare, and smart cities are all enabled by RL. However, despite the significant advances over the last few decades, a major obstacle in applying RL in practice is the lack of “safety'' guarantees such as robustness, resilience to tail-risks, operational constraints, etc. This is because the traditional RL only aims at maximizing cumulative reward. While it is possible to add penalties to rewards in a traditional RL algorithm to discourage unsafe actions, many safety constraints, such as chance constraints, cannot be simply treated as penalties. This project develops foundational technologies for safe RL-enabled systems based on Distributional Reinforcement Learning (DRL), which learns the optimal policy. While developing the foundation of DRL for safe learning-enabled systems, research and education are integrated by including new theories and algorithms developed in this project into their graduate-level courses. All team members have been regularly supervising undergraduate students and students from underrepresented groups. The team continues to leverage Women's Place at Ohio State University and the Women in Science and Engineering Program at Arizona State University to enhance the broader participation of women students and researchers. This project focuses on a comprehensive approach for the end-to-end safety of DRL-enabled systems. The end-to-end safety includes (i) policy safety: learn a safe policy to avoid the occurrence of catastrophic outcomes (corresponds to risk-sensitive RL); (ii) exploration safety -- learn a safe policy safely by avoiding dangerous actions during exploration/learning (corresponds to online RL); and (iii) environmental safety -- learn a policy that is robust to parametric uncertainty (environment change). This project includes four thrusts. Thrust 1 (Foundation of constrained DRL) aims to establish theoretical foundations of risk sensitive constrained DRL and focuses on policy and environmental safety. Thrust 2 (Online constrained DRL) considers safe online learning and decision-making and focuses on exploration safety and environmental safety when learning a safe DRL policy. Thrust 3 (Physics-Enhanced constrained DRL) exploits physics to enhance end-to-end safety. These three thrusts on foundational research are interdependent, but each focuses on a unique aspect of safe RL-enabled systems and addresses multiple safety notions. The fourth thrust will provide comprehensive validation with both high-fidelity simulations and real-world experiments using unmanned aerial vehicles.This research is supported by a partnership between the National Science Foundation and Open Philanthropy.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.