Collaborative Research: NRI: FND: Learning Graph Neural Networks for Multi-Object Manipulation

机器人要想在日常生活中充当无处不在的助手，就必须经常与涉及许多物体和由许多组成部分构成的物体的环境作斗争。目前的机器人研究集中在为孤立的操作任务提供解决方案，开发不容易跨任务工作的专门表示。该项目旨在使机器人能够学习通过多个传感器来表示和理解世界，并完成许多操作任务。具体来说，该项目将检查在严重混乱的环境中，需要多个不同的拾取和放置动作的任务。该项目将开发适用于机器人助手的自主操作方法。辅助机器人将对提高老年人和患有某些退行性疾病的人的生活质量产生重大影响。这些方法也适用于自然或人为灾害地区的操纵，在那里没有明确的对象模型。该项目开发的工具还可以提高机器人的感知、抓取和多步操作技能，以用于制造。凭借从原始感知数据中学习强大表征的能力，深度神经网络为解决自主机器人操作的关键感知和推理挑战提供了最有前途的框架。尽管 尽管现有的方法取得了成功，但它们在自动机器人将在自然环境中处理的各种场景中的规模很小。神经网络目前的这些局限性来自于在孤立的任务上进行训练，针对不同的问题使用不同的架构，以及无法扩展到包含不同或大量对象的复杂场景。该项目假设图神经网络提供了一个强大的框架，可以随着时间的推移对多个传感器流进行编码，为机器人提供丰富且可扩展的多对象和多任务感知和操纵表示。该项目研究了一些扩展到图形神经网络，以解决目前的限制，他们在自主操作中的使用。此外，该项目还研究了利用学习图神经网络进行操作规划和控制以及多步骤，多对象操作任务的新方法。为了训练这些大规模的图形网络表示，该项目将使用超大规模，物理精确，照片般逼真的模拟。该项目中开发的所有感知和行为生成技术将在一系列具有挑战性的现实世界操纵任务上进行实验验证。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

For robots to act as ubiquitous assistants in daily life, they must regularly contend with environments involving many objects and objects built of many constituent parts. Current robotics research focuses on providing solutions to isolated manipulation tasks, developing specialized representations that do not readily work across tasks. This project seeks to enable robots to learn to represent and understand the world from multiple sensors, across many manipulation tasks. Specifically, the project will examine tasks in heavily cluttered environments that require multiple distinct picking and placing actions. This project will develop autonomous manipulation methods suitable for use in robotic assistants. Assistive robots stand to make a substantial impact in increasing the quality of life of older adults and persons with certain degenerative diseases. These methods also apply to manipulation in natural or man-made disasters areas, where explicit object models are not available. The tools developed in this project can also improve robot perception, grasping, and multi-step manipulation skills for manufacturing.With their ability to learn powerful representations from raw perceptual data, deep neural networks provide the most promising framework to approach key perceptual and reasoning challenges underlying autonomous robot manipulation. Despite​ ​their success, existing approaches scale poorly to the diverse set of scenarios autonomous robots will handle in natural environments. These current limitations of neural networks arise from being trained on isolated tasks, use of different architectures for different problems, and inability to scale to complex scenes containing a varying or large number of objects. This project hypothesizes that graph neural networks provide a powerful framework that can encode multiple sensor streams over time to provide robots with rich and scalable representations for multi-object and multi-task perception and manipulation. This project examines a number of extensions to graph neural networks in order to address current limitations for their use in autonomous manipulation. Furthermore this project examines novel ways of leveraging learned graph neural networks for manipulation planning and control in clutter and for multi-step, multi-object manipulation tasks. In order to train these large-scale graph net representations this project will use extremely large scale, physically accurate, photo-realistic simulation. All perceptual and behavior generation techniques developed in this project will be experimentally validated on a set of challenging real-world manipulation tasks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.