CPS: Medium: Robust Learning for Perception-Based Autonomous Systems

考虑两个未来的自主系统用例：(I)一辆拆弹漫游车被送进一个陌生的、GPS和通信被禁止的环境(例如，洞穴或地雷)，任务是定位和拆除简易爆炸装置，以及(Ii)一架自主竞赛无人机，参加未来无人机竞赛联盟的自主化身。这两个系统都将根据简单的单输出传感设备(如惯性测量单元)和复杂的高维输出传感设备(如相机和激光雷达)的组合做出决定。从只依赖简单的单一输出传感设备到结合丰富复杂的感知传感模式的系统的转变，需要重新考虑安全关键的自主系统的设计，特别是考虑到机器和深度学习在现代感知传感器设计中扮演着不可或缺的角色。然而，这两个鼓舞人心的例子提出了一个更根本的问题：鉴于动态、环境、目标和安全/风险约束的巨大差异，这两个系统是否应该拥有具有不同特性的感知传感器？事实上，由于拆弹任务的极端安全关键性质，强调稳健性、风险规避和安全似乎是必要的。相反，无人机赛车手的设计者可能愿意牺牲健壮性，以最大限度地提高响应性和降低圈速。需求的这种极端多样性突出了需要一种原则性的方法来在这个复杂的设计空间中进行权衡，这就是本提案试图开发的。现有的设计感知/行动管道的方法要么是模块化的，通常忽略不确定性并限制组件之间的交互，要么是单一的端到端的方法，难以解释、故障排除和具有较高的样本复杂性。该项目提出了一种替代方法，并重新思考了使用机器学习和计算机视觉来处理丰富的高维感知数据的科学基础，以用于安全关键的网络物理控制应用。推进器将发展感知、规划和控制之间的集成，从而允许它们的共同设计和共同优化。对于感知表示和预测模型，使用新的稳健学习方法来表征稳健性(例如，对光照和天气变化、旋转)和性能(例如，响应性、区分性)之间的折衷，联合学习的感知地图和不确定性简档将被抽象为用于具有稳定性、性能和安全性保证的基于不确定性感知的规划和控制算法的“噪声虚拟传感器”。这些见解将被整合到新的基于感知的模型预测控制算法中，这些算法通过一个统一的基于优化的框架作用于丰富的感知数据，从而实现规划、稳定性和安全性保证。这些方法的益处将在宾夕法尼亚大学使用照片逼真模拟和配备物理相机的四轴飞行器进行实验验证，并将用于在速度/安全权衡的极端情况下演示基于感知的规划和控制算法。在教育方面，这项提案的研究成果将用于在宾夕法尼亚大学开发一系列关于安全自主、安全感知、学习和控制的课程。从长远来看，该项目的目标是创建一个新的研究人员社区，专注于基于感知的控制的稳健学习。为了实现这一目标，将利用部门的努力来增加和多样化参与该项目的博士生。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Consider two future autonomous system use-cases: (i) a bomb defusing rover sent into an unfamiliar, GPS and communication denied environment (e.g., a cave or mine), tasked with the objective of locating and defusing an improvised explosive device, and (ii) an autonomous racing drone competing in a future autonomous incarnation of the Drone Racing League. Both systems will make decisions based on inputs from a combination of simple, single output sensing devices, such as inertial measurement units, and complex, high dimensional output sensing modalities, such as cameras and LiDAR. This shift from relying only on simple, single output sensing devices to systems that incorporate rich, complex perceptual sensing modalities requires rethinking the design of safety-critical autonomous systems, especially given the inextricable role that machine and deep learning play in the design of modern perceptual sensors. These two motivating examples raise an even more fundamental question however: given the vastly different dynamics, environments, objectives, and safety/risk constraints, should these two systems have perceptual sensors with different properties? Indeed, due to the extremely safety critical nature of the bomb defusing task, an emphasis on robustness, risk aversion, and safety seems necessary. Conversely, the designer of the drone racer may be willing to sacrifice robustness to maximize responsiveness and lower lap-time. This extreme diversity in requirements highlights the need for a principled approach to navigate tradeoffs in this complex design space, which is what this proposal seeks to develop. Existing approaches to designing perception/action pipelines are either modular, which often ignore uncertainty and limit interaction between components, or monolithic and end-to-end, which are difficult to interpret, troubleshoot, and have high sample-complexity. This project proposes an alternative approach and rethinks the scientific foundations of using machine learning and computer vision to process rich high-dimensional perceptual data for use in safety-critical cyber-physical control applications. Thrusts will develop integration between perception, planning and control that allow for their co-design and co-optimization. Using novel robust learning methods for perceptual representations and predictive models that characterize tradeoffs between robustness (e.g., to lighting & weather changes, rotations) and performance (e.g., responsiveness, discriminativeness), jointly learned perception maps and uncertainty profiles will be abstracted as ``noisy virtual sensors” for use in uncertainty aware perception-based planning & control algorithms with stability, performance, and safety guarantees. These insights will be integrated into novel perception-based model predictive control algorithms, which allow for planning, stability, and safety guarantees through a unifying optimization-based framework acting on rich perceptual data. Experimental validation of the benefits of these methods will be conducted at Penn using photorealistic simulations and physical camera equipped quadcopters, and be used to demonstrate perception-based planning and control algorithms at the extremes of speed/safety tradeoffs. On the educational front, the research outcomes of this proposal will be used to develop a sequence of courses on safe autonomy, safe perception, and learning and control at the University of Pennsylvania. Longer term, the goal of this project is to create a new community of researchers that focus on robust learning for perception-based control. Towards this goal, departmental efforts will be leveraged to increase and diversify the PhD students working on this project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.