HEAP: Human-Guided Learning and Benchmarking of Robotic Heap Sorting

该项目将为基准测试、对象识别、操作和人机交互提供科学进步。我们专注于分类一堆复杂的、无结构的未知物体——类似于由一组破碎的变形体组成的核废料——作为一个极其复杂的操作任务的实例。该联盟旨在建立一个端到端的基准框架，其中包括严格的科学方法和实验工具，以应用于现实场景。基准场景将使用现成的操纵器和抓取器开发，允许创建一个经济实惠的设置，可以轻松地在物理和模拟中复制。我们将开发具有不同复杂性的基准测试场景，即抓取和推送不规则对象，从堆中抓取选定对象，识别所有对象实例并通过将它们放入相应的箱子中对对象进行排序。我们将提供可用于3D打印的对象的扫描CAD模型，以便重新创建我们的基准场景。具有现有抓取规划器和操作算法的基准将作为基线控制器实现，这些基准控制器很容易使用ROS进行交换。由于在场景理解、抓取和决策方面的挑战，机器人完全自主处理密集杂乱物或一堆未知物体的能力非常\textit{有限}。相反，我们将依靠半自主的方法，在这种方法中，人类操作员可以与系统交互（例如，使用远程操作，但不仅仅是远程操作），并给出高级命令，以补充自主技能的执行。我们系统的自治程度将根据情况的复杂性而调整。我们还将用不同的人类操作员对我们的半自主任务执行进行基准测试，并量化在自主操作中与当前SOTA的差距。在我们的半自主控制框架的基础上，我们将开发一个操作技能学习系统，该系统可以从人类操作员的演示和纠正中学习，因此可以以数据高效的方式学习复杂的操作。为了改善杂乱堆中的物体识别和分割，我们将开发新的感知算法并研究交互感知，以提高机器人在物体实例、类别和属性方面对场景的理解。

This project will provide scientific advancements for benchmarking, object recognition, manipulation and human-robot interaction. We focus on sorting a complex, unstructured heap of unknown objects --resembling nuclear waste consisting of a set of broken deformed bodies-- as an instance of an extremely complex manipulation task. The consortium aims at building an end-to-end benchmarking framework, which includes rigorous scientific methodology and experimental tools for application in realistic scenarios. Benchmark scenarios will be developed with off-the-shelf manipulators and grippers, allowing to create an affordable setup that can be easily reproduced both physically and in simulation. We will develop benchmark scenarios with varying complexities, i.e., grasping and pushing irregular objects, grasping selected objects from the heap, identifying all object instances and sorting the objects by placing them into corresponding bins. We will provide scanned CAD models of the objects that can be used for 3D printing in order to recreate our benchmark scenarios. Benchmarks with existing grasp planners and manipulation algorithms will be implemented as baseline controllers that are easily exchangeable using ROS. The ability of robots to fully autonomously handle dense clutters or a heap of unknown objects has been very \textit{limited} due to challenges in scene understanding, grasping, and decision making. Instead, we will rely on semi-autonomous approaches where a human operator can interact with the system (e.g. using tele-operation but not only) and giving high-level commands to complement the autonomous skill execution. The amount of autonomy of our system will be adapted to the complexity of the situation. We will also benchmark our semi-autonomous task execution with different human operators and quantify the gap to the current SOTA in autonomous manipulation. Building on our semi-autonomous control framework, we will develop a manipulation skill learning system that learns from demonstrations and corrections of the human operator and can therefore learn complex manipulations in a data-efficient manner. To improve object recognition and segmentation in cluttered heaps, we will develop new perception algorithms and investigate interactive perception in order to improve the robot's understanding of the scene in terms of object instances, categories and properties.