NRI: Integrating Perception and Manipulation of Deformable Objects by Learning Implicit Representations

这个国家机器人计划3.0项目的目标是开发模型和算法，通过集成视觉和触觉进行丰富的接触交互，使机器人能够自主操作可变形的物体。在许多日常任务中，从烹饪（例如，使用兼容的刮刀刮锅）到制造(例如，涉及柔性部件和材料（如复合材料和电缆）的组装任务，对可变形物体的操作是必不可少的。这些任务需要机器人执行复杂的机动，使可变形的物体接触，并施加力来弯曲和塑造它们以完成任务。协作机器人（例如，Franka Emika Panda和Kuka LBR iiwa机器人）的最新进展使机器人对可变形物体的操作在物理上成为可能，但机器人仍然缺乏执行烹饪、清洁和柔性制造等实际任务所需的模型和算法。该项目将通过其在家庭辅助和制造机器人方面的应用产生广泛的社会影响。鲁棒操纵可变形结构的能力是实现智能机器人助手的重要先导技术。机器人及其辅助应用有可能激励孩子们在STEM领域从事职业，并满足美国日益增长的辅助和制造机器人行业的需求。研究活动与教育的整合将强调积极地让本科生参与研究活动，并在本科和研究生课程中引入新的讲座材料和项目。此外，将特别强调从代表性不足的群体中招收合格的学生。本项目将开发通过新颖的隐式函数表示来感知和操纵弹性可变形物体的综合方法。弹性物体在人们的生活中无处不在，从厨房里的刮刀和海绵，到生产地板上的弹性棒，再到手术工具和组织。它们的操作是实用机器人系统的基本技能。提出的综合方法汇集了计算机视觉和机器人技术的方法，以解决对机器人应用必不可少的无缝视觉-触觉推理和行为的需求。具体来说，这种方法集成并扩展了隐函数学习的最新理论和计算进展，以建模、感知和智能地操纵弹性对象。三维几何的隐式表示由于其紧凑性和计算效率最近在计算机视觉中获得了关注；然而，它们还没有成功地应用于机器人。对于两者的成功整合，本项目的成果是：1)隐式函数理论和三维视触觉可变形物体表示算法的推广。这解决了对动态和多模态可变形对象表示的需求，特别适合机器人；特别要精简感觉-推理-行为的管道。2)隐式状态估计算法工具。这些算法将在具有隐式表征的机器人感知模式之间架起桥梁。3)隐式控制与规划的理论和算法基础。该项目产生的工具将使实际机器人系统的可变形物体的实时闭环控制成为可能。为了进行评估，本项目将探索这些算法在辅助机器人（食品准备）和仓库物流（密集包装）中的集成。研究成果有助于包括连续介质力学、计算机视觉和学习理论在内的几个相关领域，这些领域研究了受边界条件和/或隐函数理论影响的物体变形。特别是，这种方法将提供一种灵活的、可推广的、计算效率高的方法，以替代目前使用有限元分析或粒子模型的最先进的方法。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The goal of this National Robotics Initiative 3.0 project is to develop models and algorithms that enable autonomous robotic manipulation of deformable objects by integrating sight and touch for contact-rich interactions. Manipulation of deformable objects is essential in many day-to-day tasks ranging from cooking (e.g., using a compliant spatula to scrape a wok) to manufacturing (e.g., assembly tasks involving flexible parts and materials such as composites and cables). These tasks require the robot to perform complex maneuvers to bring deformable objects into contact and apply forces to bend and shape them to achieve tasks. Recent advances in collaborative robots (e.g., the Franka Emika Panda and Kuka LBR iiwa robots) have made robotic manipulation of deformable objects physically possible but robots still lack the models and algorithms necessary to perform practical tasks such as cooking, cleaning, and flexible manufacturing. This project will have broad societal impact through its applications in in-home assistive and manufacturing robotics. The ability to robustly manipulate deformable structures is an important precursor technology towards realizing intelligent robotic assistants. Robotics and their assistive applications have the potential to inspire children to pursue careers in STEM fields and meet the needs of America's growing assistive and manufacturing robotics industry. Integration of the research activities with education will emphasize actively involving undergraduates in research activities and introducing new lecture material and projects into undergraduate and graduate courses. Also, special emphasis will be given to recruit qualified students from under-represented groups.This project will develop integrated methods for perception and manipulation of elastically deformable objects via novel implicit function representations. Elastic objects are ubiquitous in people's lives, from spatulas and sponges in the kitchens to elastic rods on manufacturing floors to surgical tools and tissue. Their manipulation is an essential skill for practical robotic systems. The proposed integrative approach brings together methods from computer vision and robotics to address the need for seamless visio-tactile reasoning and acting that is essential for robotics applications. Specifically, this approach integrates and extends recent theoretical and computational advances in implicit function learning to model, perceive, and intelligently manipulate elastic objects. Implicit representations of 3D geometries have recently gained traction in computer vision owing to their compactness and computational efficiency; however, they have yet to be deployed successfully for robotics. Towards their successful integration, the outcomes of this project are: 1) A generalization of implicit function theory and algorithms for 3D visio-tactile deformable object representations. This addresses the need for dynamic and multi-modal deformable object representations particularly suited for robotics; specially to streamline the sense-reason-act pipeline. 2) Algorithmic tools for implicit state-estimation. These algorithms will bridge inherently uncertain robotic sensing modalities with implicit representations. 3) Theoretic and algorithmic foundations for implicit control and planning. The resulting tools from this project will enable real-time closed-loop control of deformable objects for practical robotic systems. For evaluation, this project will explore the integration of these algorithms for assistive robotics (food preparation) and warehouse logistics (dense packing). The research outcomes contribute to several related fields including continuum mechanics, computer vision, and learning theory where object deformations subject to boundary conditions and/or implicit function theory are studied. In particular, this approach will offer a flexible, generalizable, and computationally efficient alternative to the current state-of-the-art methods using finite element analysis or particle models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.