DeepMix: mobility-aware, lightweight, and hybrid 3D object detection for headsets
DeepMix: mobility-aware, lightweight, and hybrid 3D object detection for headsets
复制标题
DOI：
10.1145/3498361.3538945
复制
发表时间：
2022-01
期刊：
影响因子：
--
通讯作者：
Yongjie Guan;Xueyu Hou;Na Wu;Bo Han;Tao Han
中科院分区：
文献类型：
--
作者：
Yongjie Guan;Xueyu Hou;Na Wu;Bo Han;Tao Han
移动耳机应能够理解3D物理环境，以便为增强/混合现实(AR/MR)提供真正身临其境的体验。然而，它们的小尺寸和有限的计算资源使得实时3D视觉算法的执行具有极大的挑战性，众所周知，3D视觉算法比2D算法更耗费计算。在本文中，我们提出了DeepMix，一个移动性感知的，轻量级的，混合的3D目标检测框架，以改善AR/MR在移动耳机上的用户体验。受我们对最先进的3D物体检测模型的分析和评估的启发，DeepMix智能地结合了边缘辅助2D物体检测和利用耳机捕获的深度数据的新颖的设备内3D边界框估计。这导致了较低的端到端延迟，并显著提高了移动场景中的检测准确率。DeepMix的一个独特功能是，它充分利用耳机的移动性来微调检测结果，并提高检测精度。据我们所知，DeepMix是第一个达到30FPS(即端到端延迟远低于交互式AR/MR的100ms的严格要求)的3D对象检测。我们在Microsoft HoloLens上实现了一个DeepMix原型，并通过广泛的对照实验和30多个参与者的用户研究来评估其性能。与使用现有3D对象检测模型的基准相比，DeepMix不仅将检测准确率提高了9.1-37.3%，而且还将端到端延迟减少了2.68-9.15倍。
Mobile headsets should be capable of understanding 3D physical environments to offer a truly immersive experience for augmented/mixed reality (AR/MR). However, their small form-factor and limited computation resources make it extremely challenging to execute in real-time 3D vision algorithms, which are known to be more compute-intensive than their 2D counterparts. In this paper, we propose DeepMix, a mobility-aware, lightweight, and hybrid 3D object detection framework for improving the user experience of AR/MR on mobile headsets. Motivated by our analysis and evaluation of state-of-the-art 3D object detection models, DeepMix intelligently combines edge-assisted 2D object detection and novel, on-device 3D bounding box estimations that leverage depth data captured by headsets. This leads to low end-to-end latency and significantly boosts detection accuracy in mobile scenarios. A unique feature of DeepMix is that it fully exploits the mobility of headsets to fine-tune detection results and boost detection accuracy. To the best of our knowledge, DeepMix is the first 3D object detection that achieves 30 FPS (i.e., an end-to-end latency much lower than the 100 ms stringent requirement of interactive AR/MR). We implement a prototype of DeepMix on Microsoft HoloLens and evaluate its performance via both extensive controlled experiments and a user study with 30+ participants. DeepMix not only improves detection accuracy by 9.1--37.3% but also reduces end-to-end latency by 2.68--9.15×, compared to the baseline that uses existing 3D object detection models.