Expressive ASL Recognition using Millimeter-wave Wireless Signals
Expressive ASL Recognition using Millimeter-wave Wireless Signals
复制标题
DOI：
10.1109/secon48991.2020.9158441
复制
发表时间：
2020-06
期刊：
影响因子：
--
通讯作者：
P. Santhalingam;Yuanqi Du;Riley Wilkerson;Al Amin Hosain;Ding Zhang;Parth H. Pathak;H. Rangwala;R. Kushalnagar
中科院分区：
文献类型：
--
作者：
P. Santhalingam;Yuanqi Du;Riley Wilkerson;Al Amin Hosain;Ding Zhang;Parth H. Pathak;H. Rangwala;R. Kushalnagar
在美国，超过50万人使用美国手语(ASL)作为他们的主要交流方式。自动ASL识别将使聋人和重听(DHH)用户能够与不熟悉ASL的其他人以及语音控制的数字助理(例如Alexa、Siri等)交互。虽然ASL识别已经得到了广泛的研究，但对ASL非人工体标的识别却很少有人关注。非手动标记通常通过头部、躯干和肩部的运动来表达，并为签名的句子增加基本的意义和上下文。在这项工作中，我们介绍了一个基于毫米波雷达的语句级ASL识别系统ExASL。ExASL可以识别手动标记(手势)和非手动标记(头部和躯干运动)。它利用多距离聚类来识别身体部位，并对毫米波点云进行聚类。在此基础上，我们提出了一种多视角深度学习算法，该算法可以从聚类的身体部位表示中学习，从而实现富有表现力的句子级识别。我们的测试结果表明，ExASL能够识别ASL句子，单词错误率为0.79%，句子错误率为1.25%，非人工标记的准确率为83.5%。
Over half a million people in the United States use American Sign Language (ASL) as their primary mode of communication. Automatic ASL recognition would enable Deaf and Hard of Hearing (DHH) users to interact with others who are not familiar with ASL as well as voice-controlled digital assistants (e.g., Alexa, Siri, etc.). While ASL recognition has been extensively studied, there is a little attention given to recognition of ASL non-manual body markers. The non-manual markers are typically expressed through head, torso and shoulder movements, and add essential meaning and context to the signed sentences. In this work, we present ExASL, a sentence-level ASL recognition system using millimeter-wave radars. ExASL can recognize manual markers (hand gestures) and non-manual markers (head and torso movements). It utilizes multi-distance clustering to recognize body parts and cluster mmWave point clouds. We then present a multi-view deep learning algorithm that can learn from clustered body part representation for an expressive sentence-level recognition. Our evaluation shows that ExASL can recognize ASL sentences with a word error rate of 0.79%, sentence error rate of 1.25%, and non-manual markers with an accuracy of 83.5%.