NRI:FND: Unifying standard physics-based control with learning-based perception and action to enable safe and agile object manipulation using unmanned aerial vehicles

NRI：FND：将基于物理的标准控制与基于学习的感知和行动相结合，以使用无人机实现安全、敏捷的物体操纵

• 批准号: 1925189
• 负责人: Marin Kobilarov
• 金额: $ 74.96万
• 依托单位: Johns Hopkins University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-09-15 至 2023-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1925189&HistoricalAwards=false
• 关键词: NRI; FND; Unifying; standard; physics

Flying robots capable of object manipulation will enable new applications such as load pickup and delivery, infrastructure inspection and repair, agricultural crop management and harvesting. Currently though, aerial vehicles are limited in their agility and robustness when in close contact with their surroundings. More specifically, controlling aerial robots to interact with the natural environment requires complex models for inferring object dynamics in real-time through shape and appearance, dealing with contact and compliance, relying on complex perceptual cues such as occlusions or shadows, while at the same time ensuring safety and reliability. Currently, standard algorithms for robotic perception and control are not sufficient for such tasks. While machine learning techniques have proven powerful for vision-based perception and more recently for control in simple environments, current learning techniques are not directly suitable for agile autonomous vehicles where safety is critical and failed actions can be fatal for the robot and humans around it. To overcome these challenges, this project proposes a framework that combines standard control methods with learning-based perception and action in an integrated framework equipped with formal high-confidence guarantees on performance. The proposed methodology aims to enable autonomous vehicles to accomplish tasks that are currently impossible or infeasible to achieve with standard methods. The project will develop computational theory and algorithms that combine standard, i.e. physics and logic-based, control methods with learning-based control, implement a software framework and apply it to aerial manipulation tasks. More specifically, a fully differentiable framework will be developed that integrates components with known dynamics based on classical physical state representation and components that adapt to a given task through a learned implicit state representation that captures rich inertial and visual sensing. Then, a methodology for robust policy optimization with safety certificates will be developed based on high-fidelity stochastic models learned from robot data and then used to compute action policies in simulation using learned synthetic sensor models. The policies can be equipped with high-confidence formal bounds on performance and safety, which are validated and adapted in the real world. As a result, the robotic system can operate efficiently with guarantees on performance and safety. Finally, a fault-tolerant autonomy software framework will be implemented and the algorithms validated using three applications of aerial manipulation: object pick-up and transport in cluttered environments; remote sensor placement and infrastructure inspection; agricultural crop sampling and management. The proposed theory and methods are generally applicable to any robotic system operating in challenging environments, beyond aerial vehicles.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

能够操纵物体的飞行机器人将实现新的应用，如装载和交付、基础设施检查和维修、农作物管理和收获。然而，目前，飞行器在与周围环境密切接触时的敏捷性和鲁棒性有限。更具体地说，控制空中机器人与自然环境交互需要复杂的模型，通过形状和外观实时推断物体动态，处理接触和顺应性，依赖于复杂的感知线索，如遮挡或阴影，同时确保安全性和可靠性。目前，机器人感知和控制的标准算法不足以完成这些任务。虽然机器学习技术已经证明对于基于视觉的感知和最近在简单环境中的控制是强大的，但当前的学习技术并不直接适用于敏捷的自动驾驶车辆，其中安全性至关重要，并且失败的动作可能对机器人和周围的人类是致命的。该项目提出了一个框架，将标准控制方法与基于学习的感知和行动结合在一个综合框架中，该框架配备了对业绩的正式高置信度保证。拟议的方法旨在使自动驾驶汽车能够完成目前不可能或不可行的任务。该项目将开发计算理论和算法，将联合收割机标准（即基于物理和逻辑的控制方法）与基于学习的控制相结合，实施软件框架并将其应用于空中操纵任务。更具体地说，将开发一个完全可区分的框架，该框架将基于经典物理状态表示的已知动态组件和通过捕获丰富惯性和视觉感知的学习隐式状态表示适应给定任务的组件集成在一起。然后，将基于从机器人数据中学习的高保真随机模型开发具有安全证书的鲁棒策略优化方法，然后使用学习的合成传感器模型在仿真中计算动作策略。这些策略可以配备高置信度的性能和安全性的正式界限，这些界限在真实的世界中得到验证和调整。因此，机器人系统可以在保证性能和安全性的情况下有效地操作。最后，将实施容错自主软件框架，并使用三种空中操作应用程序验证算法：在杂乱环境中的物体拾取和运输;遥感器放置和基础设施检查;农作物采样和管理。所提出的理论和方法一般适用于任何机器人系统在具有挑战性的环境中运行，超越航空器。这个奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

项目成果

Perception-Based UAV Fruit Grasping Using Sub-Task Imitation Learning

• DOI: 10.1109/airpharo52252.2021.9571066
• 发表时间: 2021-10
• 期刊: 2021 Aerial Robotic Systems Physically Interacting with the Environment (AIRPHARO)
• 作者: Gabriel Baraban;Siddharth Kothiyal;Marin Kobilarov

Learn Proportional Derivative Controllable Latent Space from Pixels

• DOI: 10.1109/case49997.2022.9926535
• 发表时间: 2021-10
• 期刊: 2022 IEEE 18th International Conference on Automation Science and Engineering (CASE)
• 作者: Weiyao Wang;Marin Kobilarov;Gregory Hager

Improving the Reliability of Pick-and-Place With Aerial Vehicles Through Fault-Tolerant Software and a Custom Magnetic End-Effector

通过容错软件和定制磁性末端执行器提高飞行器取放的可靠性

• DOI: 10.1109/lra.2021.3093864
• 发表时间: 2021
• 期刊: IEEE Robotics and Automation Letters
• 影响因子: 5.2
• 作者: Garimella, Gowtham;Sheckells, Matthew;Kim, Soowon;Baraban, Gabriel;Kobilarov, Marin
• 通讯作者: Kobilarov, Marin

Robust Policy Search for an Agile Ground Vehicle Under Perception Uncertainty

• DOI: 10.1109/iros51168.2021.9636552
• 发表时间: 2021-09
• 期刊: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
• 作者: S. Sefati;Subhransu Mishra;Matthew Sheckells;Kapil D. Katyal;Jin Bai;Gregory Hager;Marin Kobilarov