NRI:FND: Unifying standard physics-based control with learning-based perception and action to enable safe and agile object manipulation using unmanned aerial vehicles

能够操纵物体的飞行机器人将实现新的应用，如装载和交付、基础设施检查和维修、农作物管理和收获。然而，目前，当飞行器与周围环境密切接触时，它们的敏捷性和稳健性是有限的。更具体地说，控制空中机器人与自然环境互动需要复杂的模型，通过形状和外观实时推断物体动态，处理接触和顺应性，依赖于复杂的感知线索，如遮挡或阴影，同时确保安全性和可靠性。目前，机器人感知和控制的标准算法还不足以完成这样的任务。虽然机器学习技术在基于视觉的感知和最近在简单环境中的控制方面已经被证明是强大的，但目前的学习技术并不直接适用于敏捷的自动驾驶汽车，因为安全至关重要，失败的动作对机器人和周围的人来说可能是致命的。为了克服这些挑战，本项目提出了一个框架，该框架将标准控制方法与基于学习的感知和行动结合在一个集成框架中，并配备了对性能的正式高置信度保证。提出的方法旨在使自动驾驶汽车能够完成目前用标准方法不可能或不可行的任务。该项目将开发计算理论和算法，将标准控制方法（即基于物理和逻辑的控制方法）与基于学习的控制方法相结合，实现软件框架并将其应用于空中操纵任务。更具体地说，将开发一个完全可微分的框架，该框架集成了基于经典物理状态表示的已知动态组件和通过捕获丰富惯性和视觉感知的学习隐式状态表示适应给定任务的组件。然后，基于从机器人数据中学习到的高保真随机模型，将开发一种具有安全证书的鲁棒策略优化方法，然后使用学习到的合成传感器模型在仿真中计算行动策略。这些策略可以在性能和安全性方面配备高置信度的正式界限，并在现实世界中进行验证和调整。因此，机器人系统可以在保证性能和安全的情况下高效运行。最后，将实现一个容错自治软件框架，并使用三种空中操纵应用验证算法：混乱环境中的物体拾取和运输；遥感器放置和基础设施检查；农作物抽样和管理。所提出的理论和方法一般适用于在具有挑战性的环境中操作的任何机器人系统，而不是飞行器。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Flying robots capable of object manipulation will enable new applications such as load pickup and delivery, infrastructure inspection and repair, agricultural crop management and harvesting. Currently though, aerial vehicles are limited in their agility and robustness when in close contact with their surroundings. More specifically, controlling aerial robots to interact with the natural environment requires complex models for inferring object dynamics in real-time through shape and appearance, dealing with contact and compliance, relying on complex perceptual cues such as occlusions or shadows, while at the same time ensuring safety and reliability. Currently, standard algorithms for robotic perception and control are not sufficient for such tasks. While machine learning techniques have proven powerful for vision-based perception and more recently for control in simple environments, current learning techniques are not directly suitable for agile autonomous vehicles where safety is critical and failed actions can be fatal for the robot and humans around it. To overcome these challenges, this project proposes a framework that combines standard control methods with learning-based perception and action in an integrated framework equipped with formal high-confidence guarantees on performance. The proposed methodology aims to enable autonomous vehicles to accomplish tasks that are currently impossible or infeasible to achieve with standard methods. The project will develop computational theory and algorithms that combine standard, i.e. physics and logic-based, control methods with learning-based control, implement a software framework and apply it to aerial manipulation tasks. More specifically, a fully differentiable framework will be developed that integrates components with known dynamics based on classical physical state representation and components that adapt to a given task through a learned implicit state representation that captures rich inertial and visual sensing. Then, a methodology for robust policy optimization with safety certificates will be developed based on high-fidelity stochastic models learned from robot data and then used to compute action policies in simulation using learned synthetic sensor models. The policies can be equipped with high-confidence formal bounds on performance and safety, which are validated and adapted in the real world. As a result, the robotic system can operate efficiently with guarantees on performance and safety. Finally, a fault-tolerant autonomy software framework will be implemented and the algorithms validated using three applications of aerial manipulation: object pick-up and transport in cluttered environments; remote sensor placement and infrastructure inspection; agricultural crop sampling and management. The proposed theory and methods are generally applicable to any robotic system operating in challenging environments, beyond aerial vehicles.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.