Robot In-hand Dexterous manipulation by extracting data from human manipulation of objects to improve robotic autonomy and dexterity - InDex

人类在处理日常物品和操纵任务、学习新技能以及适应不同或复杂的环境方面表现出色。这是我们生存的基本技能，也是我们人工制品和人造设备世界的关键特征。我们熟练地使用双手的能力是通过观察其他有技能的人和我们自己一生的学习来获得的，因为我们发现如何第一手处理物体。不幸的是，与人类相比，今天的机械手仍然无法达到如此高的灵巧度，系统也不能完全理解自己的潜力。为了让机器人真正在人类世界中运作，并实现作为智能助手的期望，它们必须能够通过掌握自己的力量、技巧和微妙的能力来操纵各种各样的未知物体。为了实现机械手的这种灵活性，需要认知能力来处理现实世界中的不确定性，并将以前学到的技能推广到新的对象和任务中。此外，我们断言，编程的复杂性必须大大降低，机器人的自主性必须变得更加自然。该索引项目旨在了解人类如何进行手部物体操纵，并用灵巧的人造手复制观察到的熟练动作，融合深度强化和迁移学习的概念，以概括多个物体和任务的手部技能。此外，对以前知识的抽象和表示将是不同硬件可重现所学技能的基础。学习将使用跨多个通道的数据，这些数据将被收集、注释并组装成一个大型数据集。数据和我们的方法将与更广泛的研究界共享，以便对照基准进行测试和复制结果。更具体地说，其核心目标是：(1)建立一个提取人类操纵物体数据的多模式人工感知架构；(2)创建手持操纵任务的多模式数据集，如重新抓取、重新定位和精细重新定位；(3)开发先进的物体建模和识别系统，包括物体提供能力和抓取特性的特征，以便概括明示的信息和可能的隐性物体用法；(4)自主学习并精确模仿人类处理任务的策略；(5)在观察和执行之间建立一座桥梁，使部署能够独立于机器人架构。

Humans excel when dealing with everyday objects and manipulation tasks, learning new skills, and adapting to different or complex environments. This is a basic skill for our survival as well as a key feature in our world of artefacts and human-made devices. Our expert ability to use our hands results from a lifetime of learning by both observing other skilled humans and ourselves as we discover how to handle objects first hand. Unfortunately, today's robotic hands are still unable to achieve such a high level of dexterity in comparison to humans nor are systems entirely able to understand their own potential. In order for robots to truly operate in a human world and fulfil the expectations as intelligent assistants, they must be able to manipulate a wide variety of unknown objects by mastering their capabilities of strength, finesse and subtlety. To achieve such dexterity with robotic hands, cognitive capacity is needed to deal with uncertainties in the real world and to generalise previously learned skills to new objects and tasks. Furthermore, we assert that the complexity of programming must be greatly reduced and robot autonomy must become much more natural. The InDex project aims to understand how humans perform in-hand object manipulation and to replicate the observed skilled movements with dexterous artificial hands, merging the concepts of deep reinforcement and transfer learning to generalise in-hand skills for multiple objects and tasks. In addition, an abstraction and representation of previous knowledge will be fundamental for the reproducibility of learned skills to different hardware. Learning will use data across multiple modalities that will be collected, annotated and assembled into a large dataset. The data and our methods will be shared with the wider research community to allow testing against benchmarks and reproduction of results. More concretely, the core objectives are: (i) to build a multi-modal artificial perception architecture that extracts data of object manipulation by humans; (ii) the creation of a multimodal dataset of in-hand manipulation tasks such as regrasping, reorienting and finely repositioning; (iii) the development of an advanced object modelling and recognition system, including the characterisation of object affordances and grasping properties, in order to encapsulate both explicit information and possible implicit object usages; (iv) to autonomously learn and precisely imitate human strategies in handling tasks; and (v) to build a bridge between observation and execution, allowing deployment that is independent of the robot architecture.