NRI:FND: Unifying standard physics-based control with learning-based perception and action to enable safe and agile object manipulation using unmanned aerial vehicles

能够操纵物体的飞行机器人将使新的应用成为可能，如装载和运送、基础设施检查和维修、农业作物管理和收获。然而，目前，当飞行器与周围环境近距离接触时，它们的敏捷性和健壮性受到限制。更具体地说，控制空中机器人与自然环境交互需要复杂的模型，通过形状和外观实时推断对象动力学，处理接触和顺应性，依赖于遮挡或阴影等复杂感知线索，同时确保安全和可靠。目前，用于机器人感知和控制的标准算法不足以完成这类任务。虽然机器学习技术在基于视觉的感知和最近在简单环境中的控制方面被证明是强大的，但目前的学习技术并不直接适用于灵活的自动驾驶车辆，在这些车辆中，安全至关重要，失败的动作可能会对机器人和周围的人类造成致命的影响。为了克服这些挑战，该项目提出了一个框架，将标准控制方法与基于学习的感知和行动结合在一个综合框架中，并配备正式的高置信度绩效保证。拟议的方法旨在使自动驾驶车辆能够完成目前用标准方法不可能或不可行的任务。该项目将开发计算理论和算法，将标准的、即基于物理和逻辑的控制方法与基于学习的控制方法相结合，实现一个软件框架，并将其应用于空中操纵任务。更具体地说，将开发一个完全可区分的框架，该框架基于经典的物理状态表示将组件与已知动力学集成在一起，并通过捕获丰富的惯性和视觉感知的学习隐式状态表示来适应给定任务的组件。然后，基于从机器人数据中学习的高保真随机模型，开发了一种基于安全证书的健壮策略优化方法，并使用学习的合成传感器模型在仿真中计算动作策略。这些政策可以配备高可信的性能和安全正式界限，并在现实世界中进行验证和调整。因此，机器人系统可以在性能和安全性得到保证的情况下高效运行。最后，实现了一个容错自主软件框架，并以空中操纵的三个应用为例对算法进行了验证：杂乱环境中的物体拾取和运输；远程传感器放置和基础设施检查；农作物采样和管理。建议的理论和方法普遍适用于在具有挑战性的环境中运行的任何机器人系统，而不仅仅是飞行器。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Flying robots capable of object manipulation will enable new applications such as load pickup and delivery, infrastructure inspection and repair, agricultural crop management and harvesting. Currently though, aerial vehicles are limited in their agility and robustness when in close contact with their surroundings. More specifically, controlling aerial robots to interact with the natural environment requires complex models for inferring object dynamics in real-time through shape and appearance, dealing with contact and compliance, relying on complex perceptual cues such as occlusions or shadows, while at the same time ensuring safety and reliability. Currently, standard algorithms for robotic perception and control are not sufficient for such tasks. While machine learning techniques have proven powerful for vision-based perception and more recently for control in simple environments, current learning techniques are not directly suitable for agile autonomous vehicles where safety is critical and failed actions can be fatal for the robot and humans around it. To overcome these challenges, this project proposes a framework that combines standard control methods with learning-based perception and action in an integrated framework equipped with formal high-confidence guarantees on performance. The proposed methodology aims to enable autonomous vehicles to accomplish tasks that are currently impossible or infeasible to achieve with standard methods. The project will develop computational theory and algorithms that combine standard, i.e. physics and logic-based, control methods with learning-based control, implement a software framework and apply it to aerial manipulation tasks. More specifically, a fully differentiable framework will be developed that integrates components with known dynamics based on classical physical state representation and components that adapt to a given task through a learned implicit state representation that captures rich inertial and visual sensing. Then, a methodology for robust policy optimization with safety certificates will be developed based on high-fidelity stochastic models learned from robot data and then used to compute action policies in simulation using learned synthetic sensor models. The policies can be equipped with high-confidence formal bounds on performance and safety, which are validated and adapted in the real world. As a result, the robotic system can operate efficiently with guarantees on performance and safety. Finally, a fault-tolerant autonomy software framework will be implemented and the algorithms validated using three applications of aerial manipulation: object pick-up and transport in cluttered environments; remote sensor placement and infrastructure inspection; agricultural crop sampling and management. The proposed theory and methods are generally applicable to any robotic system operating in challenging environments, beyond aerial vehicles.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.