Collaborative Research: SLES: Bridging offline design and online adaptation in safe learning-enabled systems

由于环境、系统目标和系统学习组件的不确定性，维护在未知环境中导航的学习型系统的安全性是一项重大挑战。该项目提出了一种新的方法来缓解这些不确定性。通过闭合在线安全监测和离线设计之间的循环，使用数据收集作为连接这两个阶段的使能模式，可以定义有意义的离线和在线安全概念。要实现该项目的技术目标，需要在使用动态分发生成的流数据表示、表征和核算支持学习的组件中的不确定性方面取得重大进展。该项目通过首先开发新的安全丰富的数据增强和域随机化技术来应对这些挑战，以培训安全的学习系统。该项目还试图确定要收集的安全丰富数据的正确类型，以确保系统的端到端安全，并使用这些数据培训支持学习的组件。这些数据生成和增强技术被集成到具有强大安全保证的新型安全意识稳健学习、控制和验证方法中。最后，该项目旨在开发在线安全监测、不确定性量化和适应技术，以应对部署期间的未知未知因素。要实现这些目标，需要以保角预测和主动学习为基础的新技术，允许在系统安全风险和主动数据收集和学习之间进行原则上的权衡，从而结束设计和部署循环。项目成果被纳入宾夕法尼亚大学和加州大学伯克利分校的本科生和研究生课程，研究团队计划在主要控制、机器学习和网络物理系统会议上组织研讨会，以帮助培养支持安全学习的系统研究人员的新型跨学科社区。

Maintaining the safety of a learning-enabled system that navigates in an unknown environment is a major challenge owing to uncertainty in the environment, the system's goals, and the system's learning-enabled components. This project proposes a novel approach to mitigating these uncertainties. The project’s novelties are the development of a two-phase design and deployment process integrated into a tight feedback loop: (1) an offline design process aimed at synthesizing systems that are provably robust and resilient to known unknowns, and (2) an automated online safety monitoring phase, during which a deployed learning-enabled system seeks to detect, learn about, and adapt to unknown unknowns. By closing the loop between online safety monitoring and offline design, using data collection as the enabling modality connecting these two phases, meaningful notions of both offline and online safety can be defined. The project’s impacts include: (i) a mathematical guarantee on the end-to-end safety of the design and deployment process described above for learning-enabled systems; (ii) methods that ensure safety with respect to known unknowns during the offline design stage, and safety with respect to unknown unknowns during deployment, when possible; and (iii) techniques that identify and learn about unknown unknowns, that is, novel sources of uncertainty, so that they can be integrated into the design of future systems. Realizing the project’s technical objectives requires major advances in representing, characterizing, and accounting for uncertainty in learning-enabled components using streaming data generated from dynamic distributions. The project addresses these challenges by first developing novel safety-rich data augmentation and domain randomization techniques for the training of safe learning-enabled systems. The project also seeks to identify the correct types of safety-rich data to be collected to ensure end-to-end safety of a system with learning-enabled components trained using this data. These data generation and augmentation techniques are integrated into novel safety-aware robust learning, control, and verification methods with strong safety guarantees. Finally, the project aims to develop online safety monitoring, uncertainty quantification, and adaptation techniques for contending with unknown unknowns during deployment. Meeting these objectives requires novel techniques rooted in conformal prediction and active learning that allow for principled tradeoffs between risks to system safety and active data collection and learning, thus closing the design and deployment loop. The project outcomes are incorporated into undergraduate and graduate classes at both Penn and UC Berkeley, and the research team plans to organize workshops at major controls, machine learning, and cyber-physical systems conferences to help foster a novel interdisciplinary community of safe learning-enabled systems researchers. All members of the research team are committed to promoting diversity and inclusion within their research groups.This research is supported by a partnership between the National Science Foundation and Open Philanthropy.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.