Collaborative Research: SLES: Bridging offline design and online adaptation in safe learning-enabled systems

由于环境、系统目标和系统的学习功能组件的不确定性，维护在未知环境中导航的学习功能系统的安全性是一个重大挑战。该项目提出了一种新的方法来减轻这些不确定性。该项目的创新之处是开发了一个两阶段的设计和部署过程，该过程集成到一个紧密的反馈回路中：（1）离线设计过程，旨在合成可证明对已知未知具有鲁棒性和弹性的系统，以及（2）自动化在线安全监控阶段，在此期间，部署的支持学习的系统寻求检测，学习和适应未知的未知。 通过关闭在线安全监控和离线设计之间的循环，使用数据收集作为连接这两个阶段的启用模式，可以定义离线和在线安全的有意义的概念。该项目的影响包括：（i）对上述学习使能系统的设计和部署过程的端到端安全性的数学保证;（ii）在离线设计阶段确保已知未知数的安全性的方法，以及在可能的情况下，在部署期间确保未知未知数的安全性的方法;以及（iii）识别和了解未知的未知数（即新的不确定性来源）的技术，以便将其集成到未来系统的设计中。实现该项目的技术目标需要在使用动态分布生成的流数据表示，表征和解释支持学习的组件中的不确定性方面取得重大进展。 该项目通过首先开发新的安全性丰富的数据增强和域随机化技术来解决这些挑战，以用于安全学习系统的培训。 该项目还试图确定要收集的安全性丰富的数据的正确类型，以确保系统的端到端安全性，该系统具有使用这些数据训练的学习功能组件。这些数据生成和增强技术被集成到具有强大安全保证的新型安全感知鲁棒学习、控制和验证方法中。 最后，该项目旨在开发在线安全监测，不确定性量化和适应技术，以应对部署过程中的未知未知因素。满足这些目标需要基于共形预测和主动学习的新技术，这些技术允许在系统安全风险与主动数据收集和学习之间进行原则性权衡，从而关闭设计和部署循环。 该项目的成果被纳入宾夕法尼亚大学和加州大学伯克利分校的本科生和研究生课程，研究团队计划在主要的控制、机器学习和网络物理系统会议上组织研讨会，以帮助培养一个新的跨学科的安全学习支持系统研究人员社区。该研究团队的所有成员都致力于促进其研究小组的多样性和包容性。这项研究得到了美国国家科学基金会和开放慈善机构之间的合作伙伴关系的支持。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Maintaining the safety of a learning-enabled system that navigates in an unknown environment is a major challenge owing to uncertainty in the environment, the system's goals, and the system's learning-enabled components. This project proposes a novel approach to mitigating these uncertainties. The project’s novelties are the development of a two-phase design and deployment process integrated into a tight feedback loop: (1) an offline design process aimed at synthesizing systems that are provably robust and resilient to known unknowns, and (2) an automated online safety monitoring phase, during which a deployed learning-enabled system seeks to detect, learn about, and adapt to unknown unknowns. By closing the loop between online safety monitoring and offline design, using data collection as the enabling modality connecting these two phases, meaningful notions of both offline and online safety can be defined. The project’s impacts include: (i) a mathematical guarantee on the end-to-end safety of the design and deployment process described above for learning-enabled systems; (ii) methods that ensure safety with respect to known unknowns during the offline design stage, and safety with respect to unknown unknowns during deployment, when possible; and (iii) techniques that identify and learn about unknown unknowns, that is, novel sources of uncertainty, so that they can be integrated into the design of future systems. Realizing the project’s technical objectives requires major advances in representing, characterizing, and accounting for uncertainty in learning-enabled components using streaming data generated from dynamic distributions. The project addresses these challenges by first developing novel safety-rich data augmentation and domain randomization techniques for the training of safe learning-enabled systems. The project also seeks to identify the correct types of safety-rich data to be collected to ensure end-to-end safety of a system with learning-enabled components trained using this data. These data generation and augmentation techniques are integrated into novel safety-aware robust learning, control, and verification methods with strong safety guarantees. Finally, the project aims to develop online safety monitoring, uncertainty quantification, and adaptation techniques for contending with unknown unknowns during deployment. Meeting these objectives requires novel techniques rooted in conformal prediction and active learning that allow for principled tradeoffs between risks to system safety and active data collection and learning, thus closing the design and deployment loop. The project outcomes are incorporated into undergraduate and graduate classes at both Penn and UC Berkeley, and the research team plans to organize workshops at major controls, machine learning, and cyber-physical systems conferences to help foster a novel interdisciplinary community of safe learning-enabled systems researchers. All members of the research team are committed to promoting diversity and inclusion within their research groups.This research is supported by a partnership between the National Science Foundation and Open Philanthropy.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.