mmASL: Environment-Independent ASL Gesture Recognition Using 60 GHz Millimeter-wave Signals
mmASL: Environment-Independent ASL Gesture Recognition Using 60 GHz Millimeter-wave Signals
复制标题
DOI：
10.1145/3381010
复制
发表时间：
2020-03-01
期刊：
影响因子：
--
通讯作者：
Kushalnagar, Raja
中科院分区：
文献类型：
--
作者：
Santhalingam, Panneer Selvam;Hosain, Al Amin;Kushalnagar, Raja
在过去的几年里，亚马逊Echo和谷歌Home等家庭助理设备变得非常流行。然而，由于它们的语音控制功能，这些设备不适合聋人和听力困难（DHH）人群。鉴于在美国有超过五十万人使用美国手语（ASL）进行交流，因此需要能够识别ASL的家庭助理系统。这项工作的目标是设计一个家庭助理系统DHH用户（简称为mmASL），可以使用60 GHz毫米波无线信号进行ASL识别。mmASL有两个重要组成部分。首先，它可以使用空间频谱图执行可靠的唤醒词检测。其次，使用可扩展和可扩展的多任务深度学习模型，mmASL可以学习ASL符号的语音属性，并使用它们来准确识别ASL符号。我们在60 GHz的软件无线电平台上实现了mmASL，并使用来自15个签名者，50个ASL签名和超过12K个签名实例的大规模数据收集对其进行了评估。我们表明，mmASL是宽容的存在其他干扰用户和他们的活动，环境的变化和不同的用户位置。我们将mmASL与基于Kinect和RGB摄像头的ASL识别系统进行了比较，发现它可以实现相当的性能（87%的平均识别准确率），验证了使用60 GHz mmWave系统进行ASL识别的可行性。
Home assistant devices such as Amazon Echo and Google Home have become tremendously popular in the last couple of years. However, due to their voice-controlled functionality, these devices are not accessible to Deaf and Hard-of-Hearing (DHH) people. Given that over half a million people in the United States communicate using American Sign Language (ASL), there is a need of a home assistant system that can recognize ASL. The objective of this work is to design a home assistant system for DHH users (referred to as mmASL) that can perform ASL recognition using 60 GHz millimeter-wave wireless signals. mmASL has two important components. First, it can perform reliable wake-word detection using spatial spectrograms. Second, using a scalable and extensible multi-task deep learning model, mmASL can learn the phonological properties of ASL signs and use them to accurately recognize the ASL signs. We implement mmASL on 60 GHz software radio platform with phased array, and evaluate it using a large-scale data collection from 15 signers, 50 ASL signs and over 12K sign instances. We show that mmASL is tolerant to the presence of other interfering users and their activities, change of environment and different user positions. We compare mmASL with a well-studied Kinect and RGB camera based ASL recognition systems, and find that it can achieve a comparable performance (87% average accuracy of sign recognition), validating the feasibility of using 60 GHz mmWave system for ASL sign recognition.