Resolving hand‐object occlusion for mixed reality with joint deep learning and model optimization
Resolving hand‐object occlusion for mixed reality with joint deep learning and model optimization
复制标题
DOI：
10.1002/cav.1956
复制
发表时间：
2020-07
影响因子：
1.1
通讯作者：
Qi Feng;Hubert P. H. Shum;S. Morishima
中科院分区：
文献类型：
--
作者：
Qi Feng;Hubert P. H. Shum;S. Morishima
通过将虚拟图像叠加到真实的世界上，混合现实促进了多样化的应用，并引起了越来越多的关注。对于许多需要用户与手术模拟等工具进行交互的应用程序来说，增强物理手持对象的虚拟外观是一个关键组成部分。然而，由于复杂的手部关节和严重的手-物体遮挡，解决手-物体交互中的遮挡是一个具有挑战性的课题。传统的基于跟踪的方法受到遮挡和形状变化的强烈模糊性的限制，而基于重建的方法显示出处理动态场景的能力很差。在这篇文章中，我们提出了一种新的真实的-时间优化系统，以解决手-对象闭塞的空间重建场景估计的手关节和面具。为了获得准确的结果，我们提出了一个联合学习过程，在两个模型之间共享信息，并联合估计手部姿势和语义分割。为了促进联合学习系统并提高其在遮挡下的准确性，我们提出了一个遮挡感知的RGB-D手部数据集，通过精确的注释和逼真的外观来减轻模糊性。与文献相比，评价显示更一致的叠加，用户研究验证了更真实的体验。
By overlaying virtual imagery onto the real world, mixed reality facilitates diverse applications and has drawn increasing attention. Enhancing physical in‐hand objects with a virtual appearance is a key component for many applications that require users to interact with tools such as surgery simulations. However, due to complex hand articulations and severe hand‐object occlusions, resolving occlusions in hand‐object interactions is a challenging topic. Traditional tracking‐based approaches are limited by strong ambiguities from occlusions and changing shapes, while reconstruction‐based methods show a poor capability of handling dynamic scenes. In this article, we propose a novel real‐time optimization system to resolve hand‐object occlusions by spatially reconstructing the scene with estimated hand joints and masks. To acquire accurate results, we propose a joint learning process that shares information between two models and jointly estimates hand poses and semantic segmentation. To facilitate the joint learning system and improve its accuracy under occlusions, we propose an occlusion‐aware RGB‐D hand data set that mitigates the ambiguity through precise annotations and photorealistic appearance. Evaluations show more consistent overlays compared with literature, and a user study verifies a more realistic experience.