NRI: Integrating Perception and Manipulation of Deformable Objects by Learning Implicit Representations

NRI：通过学习隐式表示来集成可变形物体的感知和操纵

• 批准号: 2220876
• 负责人: Nima Fazeli
• 金额: $ 74.97万
• 依托单位: Regents of the University of Michigan - Ann Arbor
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-09-01 至 2026-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2220876&HistoricalAwards=false
• 关键词: NRI; Integrating; Perception; Manipulation; Deformable

The goal of this National Robotics Initiative 3.0 project is to develop models and algorithms that enable autonomous robotic manipulation of deformable objects by integrating sight and touch for contact-rich interactions. Manipulation of deformable objects is essential in many day-to-day tasks ranging from cooking (e.g., using a compliant spatula to scrape a wok) to manufacturing (e.g., assembly tasks involving flexible parts and materials such as composites and cables). These tasks require the robot to perform complex maneuvers to bring deformable objects into contact and apply forces to bend and shape them to achieve tasks. Recent advances in collaborative robots (e.g., the Franka Emika Panda and Kuka LBR iiwa robots) have made robotic manipulation of deformable objects physically possible but robots still lack the models and algorithms necessary to perform practical tasks such as cooking, cleaning, and flexible manufacturing. This project will have broad societal impact through its applications in in-home assistive and manufacturing robotics. The ability to robustly manipulate deformable structures is an important precursor technology towards realizing intelligent robotic assistants. Robotics and their assistive applications have the potential to inspire children to pursue careers in STEM fields and meet the needs of America's growing assistive and manufacturing robotics industry. Integration of the research activities with education will emphasize actively involving undergraduates in research activities and introducing new lecture material and projects into undergraduate and graduate courses. Also, special emphasis will be given to recruit qualified students from under-represented groups.This project will develop integrated methods for perception and manipulation of elastically deformable objects via novel implicit function representations. Elastic objects are ubiquitous in people's lives, from spatulas and sponges in the kitchens to elastic rods on manufacturing floors to surgical tools and tissue. Their manipulation is an essential skill for practical robotic systems. The proposed integrative approach brings together methods from computer vision and robotics to address the need for seamless visio-tactile reasoning and acting that is essential for robotics applications. Specifically, this approach integrates and extends recent theoretical and computational advances in implicit function learning to model, perceive, and intelligently manipulate elastic objects. Implicit representations of 3D geometries have recently gained traction in computer vision owing to their compactness and computational efficiency; however, they have yet to be deployed successfully for robotics. Towards their successful integration, the outcomes of this project are: 1) A generalization of implicit function theory and algorithms for 3D visio-tactile deformable object representations. This addresses the need for dynamic and multi-modal deformable object representations particularly suited for robotics; specially to streamline the sense-reason-act pipeline. 2) Algorithmic tools for implicit state-estimation. These algorithms will bridge inherently uncertain robotic sensing modalities with implicit representations. 3) Theoretic and algorithmic foundations for implicit control and planning. The resulting tools from this project will enable real-time closed-loop control of deformable objects for practical robotic systems. For evaluation, this project will explore the integration of these algorithms for assistive robotics (food preparation) and warehouse logistics (dense packing). The research outcomes contribute to several related fields including continuum mechanics, computer vision, and learning theory where object deformations subject to boundary conditions and/or implicit function theory are studied. In particular, this approach will offer a flexible, generalizable, and computationally efficient alternative to the current state-of-the-art methods using finite element analysis or particle models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

这个国家机器人计划3.0项目的目标是开发模型和算法，通过整合视觉和触觉进行接触丰富的交互，实现可变形物体的自主机器人操作。对可变形物体的操纵在许多日常任务中是必不可少的，这些任务的范围从烹饪（例如，使用顺应性刮刀刮擦炒锅）到制造（例如，涉及柔性部件和材料（如复合材料和电缆）的装配任务）。这些任务要求机器人执行复杂的操作，使可变形物体接触，并施加力使其弯曲和成形，以完成任务。协作机器人的最新进展（例如，Franka Panda和Kuka LBR iiwa机器人）已经使可变形物体的机器人操作在物理上成为可能，但是机器人仍然缺乏执行诸如烹饪、清洁和柔性制造的实际任务所必需的模型和算法。该项目将通过其在家庭辅助和制造机器人中的应用产生广泛的社会影响。鲁棒操纵可变形结构的能力是实现智能机器人助手的重要先导技术。机器人及其辅助应用有可能激励儿童在STEM领域从事职业，并满足美国不断增长的辅助和制造机器人行业的需求。研究活动与教育的整合将强调积极参与研究活动的本科生，并引入新的讲座材料和项目到本科生和研究生课程。此外，将特别重视从代表性不足的群体中招收合格的学生。该项目将开发通过新的隐函数表示来感知和操纵弹性变形物体的综合方法。弹性物体在人们的生活中无处不在，从厨房里的抹刀和海绵到生产车间的弹性杆，再到手术工具和纸巾。它们的操纵是实用机器人系统的基本技能。所提出的综合方法将计算机视觉和机器人技术的方法结合在一起，以满足对机器人应用至关重要的无缝触觉推理和行动的需求。具体来说，这种方法集成和扩展了最近的理论和计算进展，内隐函数学习模型，感知和智能操纵弹性物体。由于其紧凑性和计算效率，3D几何形状的隐式表示最近在计算机视觉中获得了吸引力;然而，它们尚未成功地用于机器人。为了实现它们的成功集成，本项目的成果是：1）将隐函数理论和算法推广到三维触觉可变形物体表示。这解决了对特别适合于机器人的动态和多模态可变形对象表示的需求;特别是为了简化感-理-行流水线。2)隐式状态估计的数学工具。这些算法将固有的不确定机器人传感模态与隐式表示相结合。3)隐式控制与规划的理论与算法基础。该项目产生的工具将使实际机器人系统的可变形物体的实时闭环控制成为可能。为了进行评估，该项目将探索这些算法在辅助机器人（食物准备）和仓库物流（密集包装）中的集成。研究成果有助于几个相关领域，包括连续介质力学，计算机视觉和学习理论，其中对象变形受边界条件和/或隐函数理论的研究。特别是，这种方法将提供一个灵活的，可推广的，计算效率高的替代目前国家的最先进的方法，使用有限元分析或粒子models.This奖项反映了NSF的法定使命，并已被认为是值得的支持，通过评估使用基金会的智力价值和更广泛的影响审查标准。

项目成果

Data-Efficient Learning of Natural Language to Linear Temporal Logic Translators for Robot Task Specification

• DOI: 10.1109/icra48891.2023.10161125
• 发表时间: 2023-03
• 期刊: 2023 IEEE International Conference on Robotics and Automation (ICRA)
• 作者: Jiayi Pan;Glen Chou;D. Berenson

Focused Adaptation of Dynamics Models for Deformable Object Manipulation

• DOI: 10.1109/icra48891.2023.10161366
• 发表时间: 2022-09
• 期刊: 2023 IEEE International Conference on Robotics and Automation (ICRA)
• 作者: P. Mitrano;A. LaGrassa;Oliver Kroemer;D. Berenson

Manipulation via Membranes: High-Resolution and Highly Deformable Tactile Sensing and Control

通过膜进行操纵：高分辨率和高变形触觉传感和控制

• 发表时间: 2022
• 期刊: Conference on Robot Learning (CoRL
• 作者: Oller, Miquel;Berenson, Dmitry;Fazeli Nima
• 通讯作者: Fazeli Nima