Leveraging the Properties of mmWave Signals for 3D Finger Motion Tracking for Interactive IoT Applications
Leveraging the Properties of mmWave Signals for 3D Finger Motion Tracking for Interactive IoT Applications
复制标题
DOI：
10.1145/3570613
复制
发表时间：
2022-12
期刊：
影响因子：
--
通讯作者：
Yilin Liu;Shijia Zhang;Mahanth K. Gowda;Srihari Nelakuditi
中科院分区：
文献类型：
--
作者：
Yilin Liu;Shijia Zhang;Mahanth K. Gowda;Srihari Nelakuditi
毫米波信号是5G和下一代无线网络的关键组成部分，也越来越多地被考虑用于感知我们周围的环境，以实现无处不在的物联网应用。在这种情况下，本文利用毫米波信号的特性来跟踪交互式物联网应用的3D手指运动。虽然传统的基于视觉的解决方案在光线不足、遮挡的情况下会出现故障，并且还会受到隐私问题的困扰，但毫米波信号在典型的遮挡和非视线条件下工作，同时保护隐私。与专注于预定义手势分类的毫米波传感的先前工作相比，这项工作执行连续的3D手指运动跟踪。为此，我们首先通过模拟和实验观察到，与镜面反射耦合的小尺寸手指不会产生稳定的毫米波反射。然而，我们做了一个有趣的观察，专注于前臂，而不是手指可以提供稳定的反射三维手指运动跟踪。激活手指的肌肉通过前臂延伸，其运动表现为前臂上的振动。通过分析前臂反射mmWave信号的相位变化，本文设计了mm 4Arm，一个跟踪3D手指运动的系统。由于高维搜索空间、复杂的振动模式、用户多样性、硬件噪声等原因，mm 4Arm利用手指运动中的解剖学约束，并将其与基于编码器-解码器和ResNets的机器学习架构融合，从而实现准确的跟踪。10个用户的系统性能评估表明，中值误差为5.73°（定位误差为4.07 mm），对多径和手位置/方向的自然变化具有鲁棒性。在非视线条件和可能遮挡前臂的衣服下，准确度也是一致的。mm 4Arm在智能手机上运行，延迟为19 ms，能耗低。
mmWave signals form a critical component of 5G and next-generation wireless networks, which are also being increasingly considered for sensing the environment around us to enable ubiquitous IoT applications. In this context, this paper leverages the properties of mmWave signals for tracking 3D finger motion for interactive IoT applications. While conventional vision-based solutions break down under poor lighting, occlusions, and also suffer from privacy concerns, mmWave signals work under typical occlusions and non-line-of-sight conditions, while being privacy-preserving. In contrast to prior works on mmWave sensing that focus on predefined gesture classification, this work performs continuous 3D finger motion tracking. Towards this end, we first observe via simulations and experiments that the small size of fingers coupled with specular reflections do not yield stable mmWave reflections. However, we make an interesting observation that focusing on the forearm instead of the fingers can provide stable reflections for 3D finger motion tracking. Muscles that activate the fingers extend through the forearm, whose motion manifests as vibrations on the forearm. By analyzing the variation in phases of reflected mmWave signals from the forearm, this paper designs mm4Arm, a system that tracks 3D finger motion. Nontrivial challenges arise due to the high dimensional search space, complex vibration patterns, diversity across users, hardware noise, etc. mm4Arm exploits anatomical constraints in finger motions and fuses them with machine learning architectures based on encoder-decoder and ResNets in enabling accurate tracking. A systematic performance evaluation with 10 users demonstrates a median error of 5.73° (location error of 4.07 mm) with robustness to multipath and natural variation in hand position/orientation. The accuracy is also consistent under non-line-of-sight conditions and clothing that might occlude the forearm. mm4Arm runs on smartphones with a latency of 19ms and low energy overhead.