A4T: Hierarchical Affordance Detection for Transparent Objects Depth Reconstruction and Manipulation
A4T: Hierarchical Affordance Detection for Transparent Objects Depth Reconstruction and Manipulation
复制标题
DOI：
10.48550/arxiv.2207.04907
复制
发表时间：
2022-07
影响因子：
5.2
通讯作者：
Jiaqi Jiang;G. Cao;Thanh-Toan Do;Shan Luo
中科院分区：
文献类型：
--
作者：
Jiaqi Jiang;G. Cao;Thanh-Toan Do;Shan Luo
透明物体在我们的日常生活中广泛使用，因此机器人需要能够处理它们。然而，透明物体会受到光反射和折射的影响，这使得获得执行处理任务所需的准确深度图变得困难。在这封信中，我们提出了一种新颖的基于可供性的框架，用于透明对象的深度重建和操作，名为 A4T。首先使用分层 AffordanceNet 来检测透明对象及其相关的可供性，这些可供性对对象不同部分的相对位置进行编码。然后，给定预测的可供性图，使用多步深度重建方法逐步重建透明物体的深度图。最后，重建的深度图用于透明对象的基于可供性的操作。为了评估我们提出的方法，我们构建了一个真实世界的数据集 TRANS-AFF，其中包含透明对象的可供性和深度图，这是同类中的第一个。大量实验表明，我们提出的方法可以预测准确的可供性图，并且与最先进的方法相比，显着提高了透明物体的深度重建，均方根误差（以米为单位）显着从 0.097 下降到 0.042。此外，我们通过对透明物体进行的一系列机器人操作实验证明了我们提出的方法的有效性。
Transparent objects are widely used in our daily lives and therefore robots need to be able to handle them. However, transparent objects suffer from light reflection and refraction, which makes it challenging to obtain the accurate depth maps required to perform handling tasks. In this letter, we propose a novel affordance-based framework for depth reconstruction and manipulation of transparent objects, named A4T. A hierarchical AffordanceNet is first used to detect the transparent objects and their associated affordances that encode the relative positions of an object's different parts. Then, given the predicted affordance map, a multi-step depth reconstruction method is used to progressively reconstruct the depth maps of transparent objects. Finally, the reconstructed depth maps are employed for the affordance-based manipulation of transparent objects. To evaluate our proposed method, we construct a real-world dataset TRANS-AFF with affordances and depth maps of transparent objects, which is the first of its kind. Extensive experiments show that our proposed methods can predict accurate affordance maps, and significantly improve the depth reconstruction of transparent objects compared to the state-of-the-art method, with the Root Mean Squared Error in meters significantly decreased from 0.097 to 0.042. Furthermore, we demonstrate the effectiveness of our proposed method with a series of robotic manipulation experiments on transparent objects.