CPS: Small: Collaborative Research: RF Sensing for Sign Language Driven Smart Environments

依赖美国手语（ASL）作为主要交流方式的聋人严重依赖技术作为辅助设备。然而，许多技术是为听力正常的个人设计的，这使聋人社区无法从进步中受益，如果设计成与美国手语兼容，实际上可以切实改善他们的生活质量。该提案旨在通过将一种新的传感模式-射频（RF）传感-集成到旨在响应ASL用户需求的智能环境中，从而改变无处不在的传感技术。RF传感器是这种应用的独特需求，因为它们是非接触式的，可以在黑暗中或穿墙操作，保护隐私，并带来有助于ASL理解的新型信息：即微多普勒特征，它反映了运动的时变速度分布。 因此，RF感测特别适合于捕获动态符号序列的快速进展，这是ASL使用的特征。 这个合作项目不仅首次从语言学角度对基于RF的运动识别进行了研究，而且还通过将运动学与深度学习相结合实现了基于物理的机器学习方法。 通过这种方式，该项目旨在1）改进ASL识别技术和聋人智能环境的设计，2）增强语言学家用于分析语言和相关认知过程的工具，3）推进专门针对RF信号分类的机器学习方法。 该项目的重点是开发信号处理算法，以利用RF传感的独特方面来理解ASL和相关的语言特征。 更具体地说，ASL识别的三个方面被认为是：预定义的ASL单词和短语的分类，基于RF传感的动态序列分割算法的设计，和日常活动的交际手语手势的区别。将研究在一维，二维和三维中可视化和表示RF数据的新方法，用于提取语言特征并作为深度神经网络的输入。 将开发新技术用于三维时变数据流的分类，合成RF数据样本的生成，这些数据样本具有改进的运动学保真度和真实性，顺序分类和分割，以及日常运动与通信签名的区别。在该项目期间进行的关键实验将产生一个独一无二的多频RF传感器网络数据集和ASL标志的Kinect（TM）传感器测量数据集，该数据集将公开提供。 该项目通过亚拉巴马聋人和盲人研究所（AIDB）和Gallaudet大学的支持和互动，直接参与聋人社区，作为需求驱动的交流和辅助技术设计方法的一部分，最终将服务于个人，专业，该项目由网络物理系统计划和既定计划共同资助，以刺激聋人社区的竞争力。研究（EPSCoR）。该奖项反映了NSF的法定使命，并被认为是值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估的支持。

Deaf individuals who rely on American Sign Language (ASL) as their primary mode of communication heavily rely on technology as an assistive device. Yet, many technologies are designed for hearing individuals, which precludes the Deaf community from benefiting from advances, which, if designed to be compatible with ASL, could in fact generate tangible improvements in their quality of life. This proposal aims at transforming ubiquitous sensing technologies through the integration of a new sensing modality - radio frequency (RF) sensing - into smart environments designed to respond to the needs of ASL users. RF sensors are uniquely desirable for this application because they are non-contact, can operate in the dark or through-the-wall, protect privacy, and bring to bear a new type of information that will aid in ASL understanding: namely, the micro-Doppler signature, which is reflective of the time-varying velocity profiles of motion. Thus, RF sensing is uniquely suited to capture the rapid progression of dynamic sign sequences that is characteristic of ASL usage. This collaborative project not only brings to bear, for the first time, a linguistic perspective to RF-based motion recognition, but also a physics-based machine learning approach achieved through integration of kinematics with deep learning. In this way, the project aims at 1) improving ASL recognition technologies and the design of smart environments for deaf individuals, 2) augmenting the tools linguists use to analyze language and related cognitive processes, and 3) advancing machine learning approaches specifically geared towards RF signal classification. The project is focused on developing signal processing algorithms for leveraging the unique aspects of RF sensing towards understanding of ASL and related linguistic features. More specifically, three aspects of ASL recognition are considered: classification of pre-defined ASL words and phrases, design of RF-sensing based dynamic sequence segmentation algorithms, and differentiation of daily activities from communicative sign language gestures. Novel ways of visualizing and representing RF data in one, two, and three dimensions will be investigated, both for extraction of linguistic features and as inputs to deep neural networks. Novel techniques will be developed for classification of three-dimensional time-varying data streams, the generation of synthetic RF data samples that have improved kinematic fidelity and realism, sequential classification and segmentation, as well as discrimination of daily motion from communicative signing. The critical experiments conducted during this project will result in a one-of-a-kind dataset of multi-frequency RF sensor network and Kinect(tm) sensor measurements of ASL signs, which will be made publicly available. The project directly engages the Deaf community through support and interaction of the Alabama Institute of Deaf and Blind (AIDB) and Gallaudet University as part of a needs-driven approach to communicative and assistive technology design, which will ultimately serve personal, professional, and educational needs of the Deaf community.This project is jointly funded by the Cyber Physical Systems Program and the Established Program to Stimulate Competitive Research (EPSCoR).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.