CPS: Medium: Robust Learning for Perception-Based Autonomous Systems

考虑两个未来的自主系统用例：(i)将炸弹拆除漫游者发送到不熟悉的，GPS和通信中断的环境（例如，洞穴或矿井），其任务是定位和拆除简易爆炸装置，以及（ii）在未来无人驾驶赛车联赛的自主版本中竞争的自主竞速无人机。这两个系统都将根据简单的单输出传感设备（如惯性测量单元）和复杂的高维输出传感模式（如摄像头和激光雷达）的组合输入做出决策。从仅依赖简单的单输出传感设备到包含丰富、复杂的感知传感模式的系统的转变，需要重新思考安全关键自主系统的设计，特别是考虑到机器和深度学习在现代感知传感器设计中发挥的不可分割的作用。然而，这两个激励例子提出了一个更基本的问题：考虑到截然不同的动态、环境、目标和安全/风险约束，这两个系统是否应该具有不同属性的感知传感器？事实上，由于拆弹任务的安全性极其关键，强调稳健性、规避风险和安全性似乎是必要的。相反，无人驾驶赛车的设计者可能愿意牺牲鲁棒性，以最大限度地提高响应能力和缩短圈速。这种需求的极端多样性强调了在这个复杂的设计空间中导航权衡的原则性方法的必要性，这正是本提案所寻求的。现有的设计感知/行动管道的方法要么是模块化的，往往忽略不确定性和限制组件之间的交互，要么是单片和端到端，这很难解释，排除故障，并且具有很高的样本复杂性。该项目提出了一种替代方法，并重新思考了使用机器学习和计算机视觉来处理丰富的高维感知数据以用于安全关键型网络物理控制应用的科学基础。推力将发展感知、规划和控制之间的整合，从而允许它们共同设计和共同优化。使用新颖的鲁棒学习方法，对感知表征和预测模型进行表征，在鲁棒性（例如，对照明和天气变化，旋转）和性能（例如，响应性，判别性）之间进行权衡，共同学习的感知图和不确定性轮廓将被抽象为“噪声虚拟传感器”，用于具有稳定性，性能和安全性保证的基于不确定性感知的规划和控制算法。这些见解将集成到新的基于感知的模型预测控制算法中，通过基于丰富感知数据的统一优化框架，实现规划、稳定性和安全性保证。这些方法的好处的实验验证将在宾夕法尼亚大学进行，使用逼真的模拟和配备四轴飞行器的物理相机，并用于在速度/安全权衡的极端情况下演示基于感知的规划和控制算法。在教育方面，本提案的研究成果将用于宾夕法尼亚大学开发一系列关于安全自主、安全感知、学习和控制的课程。从长远来看，这个项目的目标是创建一个新的研究社区，专注于基于感知的控制的鲁棒学习。为了实现这一目标，各部门将努力增加和多样化参与该项目的博士生。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Consider two future autonomous system use-cases: (i) a bomb defusing rover sent into an unfamiliar, GPS and communication denied environment (e.g., a cave or mine), tasked with the objective of locating and defusing an improvised explosive device, and (ii) an autonomous racing drone competing in a future autonomous incarnation of the Drone Racing League. Both systems will make decisions based on inputs from a combination of simple, single output sensing devices, such as inertial measurement units, and complex, high dimensional output sensing modalities, such as cameras and LiDAR. This shift from relying only on simple, single output sensing devices to systems that incorporate rich, complex perceptual sensing modalities requires rethinking the design of safety-critical autonomous systems, especially given the inextricable role that machine and deep learning play in the design of modern perceptual sensors. These two motivating examples raise an even more fundamental question however: given the vastly different dynamics, environments, objectives, and safety/risk constraints, should these two systems have perceptual sensors with different properties? Indeed, due to the extremely safety critical nature of the bomb defusing task, an emphasis on robustness, risk aversion, and safety seems necessary. Conversely, the designer of the drone racer may be willing to sacrifice robustness to maximize responsiveness and lower lap-time. This extreme diversity in requirements highlights the need for a principled approach to navigate tradeoffs in this complex design space, which is what this proposal seeks to develop. Existing approaches to designing perception/action pipelines are either modular, which often ignore uncertainty and limit interaction between components, or monolithic and end-to-end, which are difficult to interpret, troubleshoot, and have high sample-complexity. This project proposes an alternative approach and rethinks the scientific foundations of using machine learning and computer vision to process rich high-dimensional perceptual data for use in safety-critical cyber-physical control applications. Thrusts will develop integration between perception, planning and control that allow for their co-design and co-optimization. Using novel robust learning methods for perceptual representations and predictive models that characterize tradeoffs between robustness (e.g., to lighting & weather changes, rotations) and performance (e.g., responsiveness, discriminativeness), jointly learned perception maps and uncertainty profiles will be abstracted as ``noisy virtual sensors” for use in uncertainty aware perception-based planning & control algorithms with stability, performance, and safety guarantees. These insights will be integrated into novel perception-based model predictive control algorithms, which allow for planning, stability, and safety guarantees through a unifying optimization-based framework acting on rich perceptual data. Experimental validation of the benefits of these methods will be conducted at Penn using photorealistic simulations and physical camera equipped quadcopters, and be used to demonstrate perception-based planning and control algorithms at the extremes of speed/safety tradeoffs. On the educational front, the research outcomes of this proposal will be used to develop a sequence of courses on safe autonomy, safe perception, and learning and control at the University of Pennsylvania. Longer term, the goal of this project is to create a new community of researchers that focus on robust learning for perception-based control. Towards this goal, departmental efforts will be leveraged to increase and diversify the PhD students working on this project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.