Collaborative Research: SLES: Bridging offline design and online adaptation in safe learning-enabled systems

由于环境、系统目标和系统学习支持组件的不确定性，维护在未知环境中导航的学习支持系统的安全性是一项主要挑战。该项目提出了一种新的方法来减轻这些不确定性。该项目的新颖之处在于将两阶段的设计和部署过程集成到一个紧密的反馈循环中：(1)离线设计过程，旨在综合系统，证明这些系统对已知未知具有鲁棒性和弹性；(2)自动在线安全监测阶段，在此期间，部署的具有学习能力的系统寻求检测、学习和适应未知未知。通过关闭在线安全监测和离线设计之间的循环，使用数据收集作为连接这两个阶段的启用方式，可以定义离线和在线安全的有意义的概念。该项目的影响包括：(i)对上述学习型系统的设计和部署过程的端到端安全性进行数学保证；（ii）在可能的情况下，确保离线设计阶段已知未知的安全性，以及部署阶段未知未知的安全性的方法；（iii）识别和学习未知未知的技术，即不确定性的新来源，以便将它们集成到未来系统的设计中。实现项目的技术目标需要在使用动态分布生成的流数据来表示、描述和计算支持学习的组件中的不确定性方面取得重大进展。该项目首先通过开发新的安全丰富的数据增强和领域随机化技术来解决这些挑战，用于安全学习系统的培训。该项目还试图确定要收集的安全数据的正确类型，以确保系统的端到端安全，并使用这些数据训练具有学习功能的组件。这些数据生成和增强技术集成到具有强大安全保证的新颖安全意识的鲁棒学习、控制和验证方法中。最后，该项目旨在开发在线安全监测、不确定性量化和适应技术，以应对部署过程中的未知未知。为了实现这些目标，需要基于保形预测和主动学习的新技术，这些技术允许在系统安全风险与主动数据收集和学习之间进行原则性权衡，从而关闭设计和部署循环。该项目成果被纳入宾夕法尼亚大学和加州大学伯克利分校的本科和研究生课程，研究团队计划在主要控制、机器学习和网络物理系统会议上组织研讨会，以帮助建立一个由安全学习系统研究人员组成的新型跨学科社区。研究团队的所有成员都致力于促进其研究小组的多样性和包容性。这项研究得到了美国国家科学基金会和开放慈善机构的合作支持。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Maintaining the safety of a learning-enabled system that navigates in an unknown environment is a major challenge owing to uncertainty in the environment, the system's goals, and the system's learning-enabled components. This project proposes a novel approach to mitigating these uncertainties. The project’s novelties are the development of a two-phase design and deployment process integrated into a tight feedback loop: (1) an offline design process aimed at synthesizing systems that are provably robust and resilient to known unknowns, and (2) an automated online safety monitoring phase, during which a deployed learning-enabled system seeks to detect, learn about, and adapt to unknown unknowns. By closing the loop between online safety monitoring and offline design, using data collection as the enabling modality connecting these two phases, meaningful notions of both offline and online safety can be defined. The project’s impacts include: (i) a mathematical guarantee on the end-to-end safety of the design and deployment process described above for learning-enabled systems; (ii) methods that ensure safety with respect to known unknowns during the offline design stage, and safety with respect to unknown unknowns during deployment, when possible; and (iii) techniques that identify and learn about unknown unknowns, that is, novel sources of uncertainty, so that they can be integrated into the design of future systems. Realizing the project’s technical objectives requires major advances in representing, characterizing, and accounting for uncertainty in learning-enabled components using streaming data generated from dynamic distributions. The project addresses these challenges by first developing novel safety-rich data augmentation and domain randomization techniques for the training of safe learning-enabled systems. The project also seeks to identify the correct types of safety-rich data to be collected to ensure end-to-end safety of a system with learning-enabled components trained using this data. These data generation and augmentation techniques are integrated into novel safety-aware robust learning, control, and verification methods with strong safety guarantees. Finally, the project aims to develop online safety monitoring, uncertainty quantification, and adaptation techniques for contending with unknown unknowns during deployment. Meeting these objectives requires novel techniques rooted in conformal prediction and active learning that allow for principled tradeoffs between risks to system safety and active data collection and learning, thus closing the design and deployment loop. The project outcomes are incorporated into undergraduate and graduate classes at both Penn and UC Berkeley, and the research team plans to organize workshops at major controls, machine learning, and cyber-physical systems conferences to help foster a novel interdisciplinary community of safe learning-enabled systems researchers. All members of the research team are committed to promoting diversity and inclusion within their research groups.This research is supported by a partnership between the National Science Foundation and Open Philanthropy.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.