MEgATrack: Monochrome Egocentric Articulated Hand-Tracking for Virtual Reality
MEgATrack: Monochrome Egocentric Articulated Hand-Tracking for Virtual Reality
复制标题
DOI：
10.1145/3386569.3392452
复制
发表时间：
2020-07-01
影响因子：
6.2
通讯作者：
Wang, Robert
中科院分区：
文献类型：
--
作者：
Han, Shangchen;Liu, Beibei;Wang, Robert
我们提出了一个实时手部跟踪系统，以驱动虚拟和增强现实（VR/AR）体验。使用四个鱼眼单色摄像头，我们的系统生成准确和低抖动的3D手部运动在一个大的工作体积为不同的用户。我们通过提出用于检测手和估计手关键点位置的神经网络架构来实现这一点。我们的手部检测网络能够稳健地处理各种真实的世界环境。关键点估计网络利用跟踪历史来产生空间和时间一致的姿态。我们设计了可扩展的半自动化机制，使用手动注释和自动跟踪的组合来收集大量不同的地面实况数据。此外，我们引入了一种通过跟踪检测的方法，该方法增加了平滑度，同时降低了计算成本;优化的系统在PC上以60 Hz运行，在移动的处理器上以30 Hz运行。总之，这些贡献产生了一个实用的系统，用于捕获用户的手，是Oculus Quest VR头显的默认功能，为输入和社交提供动力。
We present a system for real-time hand-tracking to drive virtual and augmented reality (VR/AR) experiences. Using four fisheye monochrome cameras, our system generates accurate and low-jitter 3D hand motion across a large working volume for a diverse set of users. We achieve this by proposing neural network architectures for detecting hands and estimating hand keypoint locations. Our hand detection network robustly handles a variety of real world environments. The keypoint estimation network leverages tracking history to produce spatially and temporally consistent poses. We design scalable, semi-automated mechanisms to collect a large and diverse set of ground truth data using a combination of manual annotation and automated tracking. Additionally, we introduce a detection-by-tracking method that increases smoothness while reducing the computational cost; the optimized system runs at 60Hz on PC and 30Hz on a mobile processor. Together, these contributions yield a practical system for capturing a user's hands and is the default feature on the Oculus Quest VR headset powering input and social presence.