Collaborative Research: NRI: FND: Graph Neural Networks for Multi-Object Manipulation

为了让机器人在日常生活中充当无处不在的助手，它们必须经常与涉及许多物体和由许多组成部分组成的物体的环境作斗争。目前的机器人研究集中于为孤立的操作任务提供解决方案，开发不容易跨任务工作的专门表示法。该项目旨在使机器人能够通过多个传感器、跨越许多操作任务来学习表示和理解世界。具体地说，该项目将检查严重杂乱环境中的任务，这些环境需要多个不同的拾取和放置操作。该项目将开发适合于机器人助手使用的自主操作方法。辅助机器人将在提高老年人和患有某些退行性疾病的人的生活质量方面产生重大影响。这些方法也适用于自然或人为灾害地区的操纵，在这些地区没有明确的对象模型可用。该项目开发的工具还可以提高机器人的感知、抓取和多步操作技能。由于深度神经网络能够从原始感知数据中学习强大的表示形式，因此为解决自主机器人操作中的关键感知和推理挑战提供了最有前途的框架。尽管​​取得了成功，但现有的方法很难适应自主机器人在自然环境中处理的各种场景。神经网络目前的这些局限性源于对孤立任务的训练，针对不同问题使用不同的体系结构，以及无法扩展到包含各种或大量对象的复杂场景。这个项目假设图神经网络提供了一个强大的框架，可以随着时间的推移对多个传感器流进行编码，为机器人提供丰富且可扩展的表示，用于多对象和多任务的感知和操作。本项目研究了图形神经网络的一些扩展，以解决它们在自主操作中使用的当前限制。此外，该项目还研究了利用学习的图形神经网络在杂乱的环境中进行操作计划和控制以及多步骤、多对象操作任务的新方法。为了训练这些大规模的图形网络表示，本项目将使用超大规模、物理精确、照片级的模拟。在这个项目中开发的所有感知和行为生成技术将在一系列具有挑战性的真实世界操作任务中进行实验验证。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

For robots to act as ubiquitous assistants in daily life, they must regularly contend with environments involving many objects and objects built of many constituent parts. Current robotics research focuses on providing solutions to isolated manipulation tasks, developing specialized representations that do not readily work across tasks. This project seeks to enable robots to learn to represent and understand the world from multiple sensors, across many manipulation tasks. Specifically, the project will examine tasks in heavily cluttered environments that require multiple distinct picking and placing actions. This project will develop autonomous manipulation methods suitable for use in robotic assistants. Assistive robots stand to make a substantial impact in increasing the quality of life of older adults and persons with certain degenerative diseases. These methods also apply to manipulation in natural or man-made disasters areas, where explicit object models are not available. The tools developed in this project can also improve robot perception, grasping, and multi-step manipulation skills for manufacturing. With their ability to learn powerful representations from raw perceptual data, deep neural networks provide the most promising framework to approach key perceptual and reasoning challenges underlying autonomous robot manipulation. Despite​ ​their success, existing approaches scale poorly to the diverse set of scenarios autonomous robots will handle in natural environments. These current limitations of neural networks arise from being trained on isolated tasks, use of different architectures for different problems, and inability to scale to complex scenes containing a varying or large number of objects. This project hypothesizes that graph neural networks provide a powerful framework that can encode multiple sensor streams over time to provide robots with rich and scalable representations for multi-object and multi-task perception and manipulation. This project examines a number of extensions to graph neural networks in order to address current limitations for their use in autonomous manipulation. Furthermore this project examines novel ways of leveraging learned graph neural networks for manipulation planning and control in clutter and for multi-step, multi-object manipulation tasks. In order to train these large-scale graph net representations this project will use extremely large scale, physically accurate, photo-realistic simulation. All perceptual and behavior generation techniques developed in this project will be experimentally validated on a set of challenging real-world manipulation tasks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.