Hand-Object Contact Prediction via Motion-Based Pseudo-Labeling and Guided Progressive Label Correction
Hand-Object Contact Prediction via Motion-Based Pseudo-Labeling and Guided Progressive Label Correction
复制标题
DOI：
--
复制
发表时间：
2021-10
期刊：
影响因子：
--
通讯作者：
Takuma Yagi;Md. Tasnimul Hasan;Yoichi Sato
中科院分区：
文献类型：
--
作者：
Takuma Yagi;Md. Tasnimul Hasan;Yoichi Sato
每一次手与物体的互动都始于接触。尽管预测手和物体之间的接触状态在理解手-物体交互中是有用的，但是关于手-物体分析的现有方法已经假设交互的手和物体是已知的，并且没有详细研究。在这项研究中，我们介绍了一种基于视频的方法来预测手和物体之间的接触。具体来说，给定一个视频和一对手和物体的轨迹，我们预测每帧的二进制接触状态（接触或无接触）。然而，注释大量的手对象轨迹和接触标签是昂贵的。为了克服这个困难，我们提出了一个半监督框架，包括（i）自动收集具有基于运动的伪标签的训练数据和（ii）引导渐进式标签校正（gPLC），它用少量可信数据校正有噪声的伪标签。我们验证了我们的框架的有效性，一个新建立的基准数据集的手物体接触预测，并显示出上级性能对现有的基线方法。代码和数据可在https://github.com/takumayagi/hand_object_contact_prediction上获得。
Every hand-object interaction begins with contact. Despite predicting the contact state between hands and objects is useful in understanding hand-object interactions, prior methods on hand-object analysis have assumed that the interacting hands and objects are known, and were not studied in detail. In this study, we introduce a video-based method for predicting contact between a hand and an object. Specifically, given a video and a pair of hand and object tracks, we predict a binary contact state (contact or no-contact) for each frame. However, annotating a large number of hand-object tracks and contact labels is costly. To overcome the difficulty, we propose a semi-supervised framework consisting of (i) automatic collection of training data with motion-based pseudo-labels and (ii) guided progressive label correction (gPLC), which corrects noisy pseudo-labels with a small amount of trusted data. We validated our framework's effectiveness on a newly built benchmark dataset for hand-object contact prediction and showed superior performance against existing baseline methods. Code and data are available at https://github.com/takumayagi/hand_object_contact_prediction.