American Sign Language Recognition Using RF Sensing
American Sign Language Recognition Using RF Sensing
复制标题
DOI：
10.1109/jsen.2020.3022376
复制
发表时间：
2021-02-01
影响因子：
4.3
通讯作者：
Mdrafi, Robiulhossain
中科院分区：
文献类型：
--
作者：
Gurbuz, Sevgi Z.;Gurbuz, Ali Cafer;Mdrafi, Robiulhossain
许多人机交互技术是为听力正常的人设计的，并且依赖于发声语音，这使得聋人群体中使用美国手语（ASL）的用户无法从这些进步中受益。虽然在利用视频或可穿戴手套进行美国手语识别方面已经取得了很大进展，但家庭中使用视频引发了隐私问题，而可穿戴手套严重限制了行动并干扰了日常生活。
方法：本文提出将射频传感器用于服务聋人群体的人机交互应用。一个多频射频传感器网络被用于在不考虑光照条件的情况下，对美国手语的手势进行无创、非接触式测量。利用短时傅里叶变换进行时频分析，揭示了由于微多普勒效应而在射频数据中呈现的独特运动模式。使用机器学习（ML）研究射频美国手语数据的语言特性。
结果：通过分形复杂度测量，美国手语手势的信息含量被证明大于日常生活中其他上身活动的信息含量。这可用于区分日常活动和手语，而射频数据的特征表明，非手语使用者的模仿手语与母语为美国手语的人的手语有99%的可区分性。射频传感器网络数据的特征级融合被用于在20个母语为美国手语的手势分类中达到72.5%的准确率。
启示：射频传感可用于研究美国手语的动态语言特性，并设计以聋人为中心的智能环境，以实现对美国手语的无创、远程识别。机器学习算法应该以母语为美国手语的数据，而不是模仿数据作为基准。
Many technologies for human-computer interaction have been designed for hearing individuals and depend upon vocalized speech, precluding users of American Sign Language (ASL) in the Deaf community from benefiting from these advancements. While great strides have been made in ASL recognition with video or wearable gloves, the use of video in homes has raised privacy concerns, while wearable gloves severely restrict movement and infringe on daily life. Methods: This article proposes the use of RF sensors for HCI applications serving the Deaf community. A multi-frequency RF sensor network is used to acquire non-invasive, non-contact measurements of ASL signing irrespective of lighting conditions. The unique patterns of motion present in the RF data due to the micro-Doppler effect are revealed using time-frequency analysis with the Short-Time Fourier Transform. Linguistic properties of RF ASL data are investigated using machine learning (ML). Results: The information content, measured by fractal complexity, of ASL signing is shown to be greater than that of other upper body activities encountered in daily living. This can be used to differentiate daily activities from signing, while features from RF data show that imitation signing by non-signers is 99% differentiable from native ASL signing. Feature-level fusion of RF sensor network data is used to achieve 72.5% accuracy in classification of 20 native ASL signs. Implications: RF sensing can be used to study dynamic linguistic properties of ASL and design Deaf-centric smart environments for non-invasive, remote recognition of ASL. ML algorithms should be benchmarked on native, not imitation, ASL data.