Interactive Perception-Action-Learning for Modelling Objects

在没有详细的先前模型的情况下操纵日常物品仍然超出了现有机器人的能力。这是由于不同类型的物体带来的许多挑战：操纵需要了解物体的物理属性，如形状、质量、摩擦、弹性等，并建立准确的模型。许多物体是可变形的、铰接的，甚至是具有未定义形状(例如植物)的有机物体，因此固定的模型是不够的。最重要的是，对象可能很难感知，通常是因为场景混乱，或者复杂的照明和反射特性，如镜面反射度或部分透明度。创建如此丰富的对象表示超出了当前用于抓取和操作的数据集和基准实践的范围。在这个项目中，我们将开发一个自动交互感知管道来构建如此丰富的数字化。更具体地说，在IPALM中，我们将开发通过探索性操作自动数字化对象及其物理属性的方法。这些方法将被用来建立机器人学中真实的抓取和操纵实验所需的大量对象模型。家居物品，如工具、厨房用具、衣服和食品等，不仅广泛存在并成为许多实际应用的焦点，而且给现实场景中的机器人对象感知和操作带来了巨大的挑战。我们建议通过包括可变形、可铰接、可交互、镜面反射或透明的家用物品以及布料和食品等不可成形物品来推动最新技术的发展。我们的方法将从不同的方式同时学习感知和抓取所必需的物理属性：视觉、触摸、音频以及在线手册等文本文档，并将包括以下属性：3D模型、纹理、弹性、摩擦、重量、大小和预期用途的抓取技术。我们方法的核心是两级建模，其中类别级模型为捕获特定对象的实例级属性提供了先例。我们将利用在线可用资源来构建先前的类别级别模型，感知-动作-学习循环将使用机器人的视觉、音频和触摸来模拟实例级别的对象属性。作为回报，从新实例获得的知识将用于改进类别级知识。我们的方法将允许我们有效地为不同类型的对象创建一个大型模型数据库，这将适合于例如训练基于神经网络的方法或增强现有的模拟器。我们将提出一个对象抓取的基准和评估指标，以便与我们数据库中的各种机器人平台产生的结果进行比较。我们追求的主要目标是与商业相关的机器人技术，这得到了几家公司的支持函的支持。我们将与来自5个欧盟国家的5家世界级学术机构(伦敦帝国理工学院(英国)、波尔多大学(法国)、机器人工业研究所(西班牙)、阿尔托大学(芬兰)和捷克技术大学(捷克共和国))共同努力实现我们的目标，组建一个在获取、处理和学习具有机器人应用的多模式信息方面具有强大专业知识的互补研究团队。

Manipulating everyday objects without detailed prior models is still beyond the capabilities of existing robots. This is due to many challenges posed by diverse types of objects: Manipulation requires understanding and accurate model of physical properties of objects such as shape, mass, friction, elasticity, etc. Many objects are deformable, articulated, or even organic with undefined shape (e.g., plants) such that a fixed model is insufficient. On top of this, objects may be difficult to perceive, typically because of cluttered scenarios, or complex lighting and reflectance properties such as specularity or partial transparency. Creating such rich representations of objects is beyond current datasets and benchmarking practices used for grasping and manipulation. In this project we will develop an automated interactive perception pipeline for building such rich digitization.More specifically, in IPALM, we will develop methods for the automatic digitization of objects and their physical properties by exploratory manipulations. These methods will be used to build a large collection of object models required for realistic grasping and manipulation experiments in robotics. Household objects such as tools, kitchenware, clothes, and food items are not only widely accessible and in focus of many practical applications but also pose great challenges for robot object perception and manipulation in realistic scenarios. We propose to advance the state of the art by including household objects that can be deformable, articulated, interactive, specular or transparent, as well as shapeless such as cloth and food items. Our methods will learn physical properties essential for perception and grasping simultaneously from different modalities: vision, touch, audio as well as text documents such as online manuals and will include the following properties: 3D model, texture, elasticity, friction, weight, size and grasping techniques for intended use. At the core of our approach is a two-level modelling, where a category level model provides priors for capturing instance level attributes of specific objects. We will exploit online available resources to build prior category level models and a perception-action-learning loop will use the robot's vision, audio, and touch to model instance level object properties. In return, knowledge acquired from a new instance will be used to improve the category-level knowledge. Our approach will allow us to efficiently create a large database of models for objects of diverse types, which will be suitable for example for training neural network based methods or enhancing existing simulators. We will propose a benchmark and evaluation metrics for object grasping, to enable comparisons of results generated with various robotics platforms on our database.The main objectives we pursue are commercially relevant robotics technologies, as endorsed by the support letters of several companies. We will pursue our goals with a consortium that brings together 5 world-class academic institutions from 5 EU countries (Imperial College London (UK), University of Bordeaux (France), Institut de Robotica i informàtica Industrial (Spain), Aalto University (Finland), and the Czech Technical University (Czech Republic), assembling a complementary research team with strong expertise in the acquisition, processing and learning of multimodal information with applications in robotics.