NRI: Robust and Efficient Physics-based Learning and Reasoning in Degraded Environments

NRI：退化环境中稳健且高效的基于物理的学习和推理

• 批准号: 2132972
• 负责人: Abdeslam Boularias
• 金额: $ 149.03万
• 依托单位: Rutgers University New Brunswick
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-02-01 至 2026-01-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2132972&HistoricalAwards=false
• 关键词: NRI; Robust; Efficient; Physics; based

This project will perform fundamental research into developing and integrating physics-driven reasoning and planning techniques to enable autonomous robots to manipulate unknown irregular objects and navigate in unstructured, dynamic environments. The developed techniques will be deployed on RoboMantis: a four-legged, wheeled robot that can assist in first-response missions. The project will fill the important gap between existing research on learning models of unknown objects from data and research on developing adequate simulation tools for robotic manipulation and locomotion by answering three fundamental questions: 1) How to efficiently simulate the effects of robotic actions on objects with uncertain models? 2) How to use physics simulation tools to plan manipulation and locomotion strategies for navigating in unstructured terrains? and, 3) How to learn physical models of objects on the fly? The project builds on top of progress in computer vision, physics simulation, and planning, towards developing an efficient toolset for robotic navigation in rubble.The main technical objectives of this project are to: 1) Develop physics simulation tools that can be used for efficiently inferring models of both rigid and non-rigid objects and for robust planning, 2) Develop optimization tools for learning models of objects from limited vision and interaction data, 3) Develop manipulation and navigation algorithms that can leverage the learned models, and 4) Demonstrate the fully integrated system on a diverse range of tasks related to search and rescue operations, such as manipulating unknown objects in clutter and navigating in rubble. The project will adopt a Bayesian approach where models of objects that are typically found in piles of rubble, such as debris and rocks, will be inferred from a few RGB-D images providing partial views of the scenes, and also from their responses to forces applied by the robot during locomotion and manipulation actions. Hypotheses of various models will be used to simulate the effects of the exerted forces on the objects. Models that best reproduce the observed effects of the forces will be given the highest probabilities. The inferred models will then be used to plan robust manipulation and locomotion actions that allow the robot to clear its way and advance through a pile of debris. The project brings together an interdisciplinary team of investigators who have expertise in computer vision, physics simulation and planning. Implementations of the developed solutions will be provided to the research community as open-source software packages. This will be coupled with the generation of educational material, especially programming assignments on manipulation challenges that require physics reasoning, which will be shared with the academic community. The material will aim to attract undergraduate students early in their studies to STEM by using hands-on experience that can be provided with the use of robotics, while also exposing them to foundational methods and data-driven tools. When appropriate, efforts will be made to introduce the research, in particular the hardware demonstrations, to K-12 students to cultivate their early interests in robotics, which touches all aspects of STEM. In the process, the PIs will aim to leverage diversity programs at Rutgers University to recruit and support underrepresented groups.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目将进行基础研究，开发和集成物理驱动的推理和规划技术，使自主机器人能够操纵未知的不规则物体并在非结构化的动态环境中导航。开发的技术将部署在 RoboMantis 上：一种四足轮式机器人，可以协助执行急救任务。该项目将通过回答三个基本问题来填补现有的从数据中学习未知物体模型的研究与开发适当的机器人操纵和运动模拟工具的研究之间的重要空白：1）如何有效地模拟机器人动作对模型不确定的物体的影响？ 2）如何使用物理模拟工具来规划在非结构化地形中导航的操纵和运动策略？以及，3）如何动态学习物体的物理模型？该项目建立在计算机视觉、物理模拟和规划方面的进展之上，旨在开发用于瓦砾中机器人导航的高效工具集。该项目的主要技术目标是：1) 开发物理模拟工具，可用于有效推断刚性和非刚性物体的模型以及稳健的规划，2) 开发优化工具，用于从有限的视觉和交互数据中学习物体模型，3) 开发操纵和导航算法， 可以利用学习到的模型，以及 4) 在与搜索和救援行动相关的各种任务上展示完全集成的系统，例如在杂乱中操纵未知物体和在瓦砾中导航。该项目将采用贝叶斯方法，通常在瓦砾堆中发现的物体模型，例如碎片和岩石，将从提供部分场景视图的一些 RGB-D 图像以及它们对机器人在运动和操纵动作期间施加的力的响应中推断出来。各种模型的假设将用于模拟施加在物体上的力的效果。最能再现所观察到的力效应的模型将被赋予最高的概率。然后，推断的模型将用于规划稳健的操纵和运动动作，使机器人能够清理道路并在一堆碎片中前进。该项目汇集了一支跨学科的研究人员团队，他们在计算机视觉、物理模拟和规划方面拥有专业知识。所开发解决方案的实施将作为开源软件包提供给研究界。这将与教育材料的生成相结合，特别是需要物理推理的操纵挑战的编程作业，这些材料将与学术界共享。该材料旨在通过使用机器人技术提供的实践经验，吸引本科生在学习早期进入 STEM，同时让他们接触基础方法和数据驱动工具。在适当的时候，我们将努力向 K-12 学生介绍这项研究，特别是硬件演示，以培养他们对机器人技术的早期兴趣，因为机器人技术涉及 STEM 的各个方面。在此过程中，PI 将致力于利用罗格斯大学的多元化计划来招募和支持代表性不足的群体。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力优点和更广泛的影响审查标准进行评估，认为值得支持。

项目成果

Learning Sensorimotor Primitives of Sequential Manipulation Tasks from Visual Demonstrations

从视觉演示中学习顺序操作任务的感觉运动原语

• DOI: 10.1109/icra46639.2022.9811703
• 发表时间: 2022
• 期刊: International Conference on Robotics and Automation (ICRA
• 作者: Liang, Junchi;Wen, Bowen;Bekris, Kostas;Boularias, Abdeslam
• 通讯作者: Boularias, Abdeslam

Visual Foresight Trees for Object Retrieval From Clutter With Nonprehensile Rearrangement

• DOI: 10.1109/lra.2021.3123373
• 发表时间: 2022-01-01
• 期刊: IEEE ROBOTICS AND AUTOMATION LETTERS
• 影响因子: 5.2
• 作者: Huang, Baichuan;Han, Shuai D.;Boularias, Abdeslam
• 通讯作者: Boularias, Abdeslam

Real-time Height-field Simulation of Sand and Water Mixtures

• DOI: 10.1145/3610548.3618159
• 发表时间: 2023-12
• 期刊: SIGGRAPH Asia 2023 Conference Papers
• 作者: Haozhe Su;Siyu Zhang;Zherong Pan;Mridul Aanjaneya;Xifeng Gao;Kui Wu

Interleaving Monte Carlo Tree Search and Self-Supervised Learning for Object Retrieval in Clutter

• DOI: 10.1109/icra46639.2022.9812132
• 发表时间: 2022-02
• 期刊: 2022 International Conference on Robotics and Automation (ICRA)
• 作者: Baichuan Huang;Teng Guo;Abdeslam Boularias;Jingjin Yu

Learning Category-Level Manipulation Tasks from Point Clouds with Dynamic Graph CNNs

• DOI: 10.1109/icra48891.2023.10160820
• 发表时间: 2022-09
• 期刊: 2023 IEEE International Conference on Robotics and Automation (ICRA)
• 作者: Junchi Liang;Abdeslam Boularias