Interactive Perception-Action-Learning for Modelling Objects

在没有详细的先验模型的情况下操纵日常物体仍然超出了现有机器人的能力。这是由于不同类型的物体所带来的许多挑战：操纵需要理解物体的物理特性（如形状、质量、摩擦力、弹性等）并对其进行准确建模。#21453;，这样的固定模式是不够的。除此之外，对象可能难以感知，通常是因为混乱的场景，或复杂的照明和反射特性，如镜面反射或部分透明。创建如此丰富的对象表示超出了当前用于抓取和操纵的数据集和基准测试实践。在这个项目中，我们将开发一个自动化的交互式感知管道来构建这样丰富的数字化。更具体地说，在IPALM中，我们将开发通过探索性操作自动数字化对象及其物理属性的方法。这些方法将被用来建立一个大的收集所需的现实把握和机器人操作实验的对象模型。工具、厨具、衣服和食品等家用物品不仅广泛使用并成为许多实际应用的焦点，而且对现实场景中的机器人物体感知和操纵构成了巨大挑战。我们建议通过包括可以变形的、铰接的、交互的、镜面的或透明的以及不成形的（例如布料和食品）的家居对象来推进现有技术。我们的方法将同时从不同的模式中学习感知和抓握所必需的物理特性：视觉，触觉，音频以及在线手册等文本文档，并将包括以下属性：3D模型，纹理，弹性，摩擦力，重量，尺寸和预期用途的抓握技术。在我们的方法的核心是一个两级建模，其中类别级模型提供了用于捕获特定对象的实例级属性的先验。我们将利用在线可用资源来构建先前的类别级别模型，感知-动作-学习循环将使用机器人的视觉，音频和触觉来建模实例级别的对象属性。作为回报，从新实例获得的知识将用于改进类别级知识。我们的方法将使我们能够有效地为不同类型的对象创建一个大型模型数据库，这将适用于例如训练基于神经网络的方法或增强现有的模拟器。我们将提出物体抓取的基准和评估指标，以便在我们的数据库中比较各种机器人平台生成的结果。我们追求的主要目标是商业相关的机器人技术，并得到多家公司的支持信的认可。我们将与来自5个欧盟国家的5个世界级学术机构一起追求我们的目标（帝国理工学院伦敦（英国）、波尔多大学（法国）、机器人与信息工业学院（西班牙）、阿尔托大学（芬兰）和捷克技术大学（捷克共和国），组建了一个互补的研究团队，在获取，处理和学习多模态信息与机器人应用方面具有很强的专业知识。

Manipulating everyday objects without detailed prior models is still beyond the capabilities of existing robots. This is due to many challenges posed by diverse types of objects: Manipulation requires understanding and accurate model of physical properties of objects such as shape, mass, friction, elasticity, etc. Many objects are deformable, articulated, or even organic with undefined shape (e.g., plants) such that a fixed model is insufficient. On top of this, objects may be difficult to perceive, typically because of cluttered scenarios, or complex lighting and reflectance properties such as specularity or partial transparency. Creating such rich representations of objects is beyond current datasets and benchmarking practices used for grasping and manipulation. In this project we will develop an automated interactive perception pipeline for building such rich digitization.More specifically, in IPALM, we will develop methods for the automatic digitization of objects and their physical properties by exploratory manipulations. These methods will be used to build a large collection of object models required for realistic grasping and manipulation experiments in robotics. Household objects such as tools, kitchenware, clothes, and food items are not only widely accessible and in focus of many practical applications but also pose great challenges for robot object perception and manipulation in realistic scenarios. We propose to advance the state of the art by including household objects that can be deformable, articulated, interactive, specular or transparent, as well as shapeless such as cloth and food items. Our methods will learn physical properties essential for perception and grasping simultaneously from different modalities: vision, touch, audio as well as text documents such as online manuals and will include the following properties: 3D model, texture, elasticity, friction, weight, size and grasping techniques for intended use. At the core of our approach is a two-level modelling, where a category level model provides priors for capturing instance level attributes of specific objects. We will exploit online available resources to build prior category level models and a perception-action-learning loop will use the robot's vision, audio, and touch to model instance level object properties. In return, knowledge acquired from a new instance will be used to improve the category-level knowledge. Our approach will allow us to efficiently create a large database of models for objects of diverse types, which will be suitable for example for training neural network based methods or enhancing existing simulators. We will propose a benchmark and evaluation metrics for object grasping, to enable comparisons of results generated with various robotics platforms on our database.The main objectives we pursue are commercially relevant robotics technologies, as endorsed by the support letters of several companies. We will pursue our goals with a consortium that brings together 5 world-class academic institutions from 5 EU countries (Imperial College London (UK), University of Bordeaux (France), Institut de Robotica i informàtica Industrial (Spain), Aalto University (Finland), and the Czech Technical University (Czech Republic), assembling a complementary research team with strong expertise in the acquisition, processing and learning of multimodal information with applications in robotics.