CPS: Small: Collaborative Research: RF Sensing for Sign Language Driven Smart Environments

CPS：小型：协作研究：手语驱动智能环境的射频传感

• 批准号: 1931861
• 负责人: Ali Gurbuz
• 金额: $ 13.3万
• 依托单位: Mississippi State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-10-01 至 2022-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1931861&HistoricalAwards=false
• 关键词: CPS; Small; Collaborative; Research; RF

Deaf individuals who rely on American Sign Language (ASL) as their primary mode of communication heavily rely on technology as an assistive device. Yet, many technologies are designed for hearing individuals, which precludes the Deaf community from benefiting from advances, which, if designed to be compatible with ASL, could in fact generate tangible improvements in their quality of life. This proposal aims at transforming ubiquitous sensing technologies through the integration of a new sensing modality - radio frequency (RF) sensing - into smart environments designed to respond to the needs of ASL users. RF sensors are uniquely desirable for this application because they are non-contact, can operate in the dark or through-the-wall, protect privacy, and bring to bear a new type of information that will aid in ASL understanding: namely, the micro-Doppler signature, which is reflective of the time-varying velocity profiles of motion. Thus, RF sensing is uniquely suited to capture the rapid progression of dynamic sign sequences that is characteristic of ASL usage. This collaborative project not only brings to bear, for the first time, a linguistic perspective to RF-based motion recognition, but also a physics-based machine learning approach achieved through integration of kinematics with deep learning. In this way, the project aims at 1) improving ASL recognition technologies and the design of smart environments for deaf individuals, 2) augmenting the tools linguists use to analyze language and related cognitive processes, and 3) advancing machine learning approaches specifically geared towards RF signal classification. The project is focused on developing signal processing algorithms for leveraging the unique aspects of RF sensing towards understanding of ASL and related linguistic features. More specifically, three aspects of ASL recognition are considered: classification of pre-defined ASL words and phrases, design of RF-sensing based dynamic sequence segmentation algorithms, and differentiation of daily activities from communicative sign language gestures. Novel ways of visualizing and representing RF data in one, two, and three dimensions will be investigated, both for extraction of linguistic features and as inputs to deep neural networks. Novel techniques will be developed for classification of three-dimensional time-varying data streams, the generation of synthetic RF data samples that have improved kinematic fidelity and realism, sequential classification and segmentation, as well as discrimination of daily motion from communicative signing. The critical experiments conducted during this project will result in a one-of-a-kind dataset of multi-frequency RF sensor network and Kinect(tm) sensor measurements of ASL signs, which will be made publicly available. The project directly engages the Deaf community through support and interaction of the Alabama Institute of Deaf and Blind (AIDB) and Gallaudet University as part of a needs-driven approach to communicative and assistive technology design, which will ultimately serve personal, professional, and educational needs of the Deaf community.This project is jointly funded by the Cyber Physical Systems Program and the Established Program to Stimulate Competitive Research (EPSCoR).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

依赖美国手语（ASL）作为主要交流方式的聋人严重依赖技术作为辅助设备。然而，许多技术都是为听力正常的人设计的，这就排除了聋人社区从进步中受益的可能性，如果这些技术被设计成与美国手语兼容，实际上可以为他们的生活质量带来切实的改善。该提案旨在通过将一种新的传感方式-射频（RF）传感-集成到旨在响应美国手语用户需求的智能环境中，从而转变无处不在的传感技术。射频传感器在这种应用中是独一无二的，因为它们是非接触式的，可以在黑暗中或穿墙操作，保护隐私，并带来一种有助于美国手语理解的新型信息：即微多普勒特征，它反映了运动随时间变化的速度曲线。因此，射频传感是唯一适合捕捉动态符号序列的快速发展，这是美国手语使用的特点。该合作项目不仅首次为基于射频的运动识别带来了语言学视角，而且还通过运动学与深度学习的集成实现了基于物理的机器学习方法。通过这种方式，该项目旨在1)改进美国手语识别技术和为聋哑人设计智能环境，2)增强语言学家用于分析语言和相关认知过程的工具，以及3)推进专门针对射频信号分类的机器学习方法。该项目专注于开发信号处理算法，以利用射频传感的独特方面来理解美国手语和相关语言特征。更具体地说，本文考虑了手语识别的三个方面：预定义的ASL单词和短语的分类，基于rf传感的动态序列分割算法的设计，以及日常活动与交际手势的区分。将研究在一、二、三维中可视化和表示射频数据的新方法，以提取语言特征和作为深度神经网络的输入。将开发新的技术用于三维时变数据流的分类，合成射频数据样本的生成，这些数据样本提高了运动保真度和真实感，顺序分类和分割，以及从交际手势中区分日常运动。在该项目中进行的关键实验将产生一个独一无二的多频射频传感器网络和Kinect传感器测量ASL标志的数据集，该数据集将公开提供。该项目通过阿拉巴马聋人与盲人研究所（AIDB）和加劳德特大学的支持和互动，直接吸引聋人社区参与，作为需求驱动的交流和辅助技术设计方法的一部分，最终将满足聋人社区的个人、专业和教育需求。该项目由网络物理系统计划和促进竞争研究的既定计划（EPSCoR）联合资助。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Data Driven Learning of Constrained Measurement Matrices for Signal Reconstruction

用于信号重建的约束测量矩阵的数据驱动学习

• DOI: 10.1109/ieeeconf53345.2021.9723098
• 发表时间: 2021
• 期刊: and Computers
• 作者: Mdrafi, Robiulhossain;Gurbuz, Ali Cafer
• 通讯作者: Gurbuz, Ali Cafer

Compressed Classification from Learned Measurements

• DOI: 10.1109/iccvw54120.2021.00449
• 发表时间: 2021-10
• 期刊: 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)
• 作者: Robiulhossain Mdrafi;A. Gürbüz

ASL Recognition Based on Kinematics Derived from a Multi-Frequency RF Sensor Network

基于多频射频传感器网络运动学的 ASL 识别

• DOI: 10.1109/sensors47125.2020.9278864
• 发表时间: 2020
• 期刊: IEEE Sensors Conference
• 作者: Gurbuz, Sevgi Z.;Gurbuz, Ali C.;Malaia, Evie A.;Griffin, Darrin J.;Crawford, Chris;Kurtoglu, Emre;Rahman, M. Mahbubur;Aksu, Ridvan;Mdrafi, Robiulhossain
• 通讯作者: Mdrafi, Robiulhossain

Sequential Classification of ASL Signs in the Context of Daily Living Using RF Sensing

• DOI: 10.1109/radarconf2147009.2021.9455178
• 发表时间: 2021-05
• 期刊: 2021 IEEE Radar Conference (RadarConf21)
• 作者: Emre Kurtoğlu;A. Gurbuz;E. Malaia;Darrin J. Griffin;Chris S. Crawford;S. Gurbuz

Complexity in sign languages

• DOI: 10.1515/lingvan-2021-0005
• 发表时间: 2022-10-31
• 期刊: LINGUISTICS VANGUARD
• 影响因子: 1.1
• 作者: Malaia, Evie A.;Borneman, Joshua D.;Gurbuz, Ali C.
• 通讯作者: Gurbuz, Ali C.