CAREER: Sign-to-Speech: An Edge-IoT Platform and Software Library for Real Time Sign Language Recognition

职业：手语转语音：用于实时手语识别的边缘物联网平台和软件库

• 批准号: 2046972
• 负责人: Mahanth Gowda
• 金额: $ 50万
• 依托单位: Pennsylvania State Univ University Park
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-03-01 至 2026-02-28
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2046972&HistoricalAwards=false
• 关键词: CAREER; Sign; Speech; Edge; IoT

This project will advance the state-of-the-art in cross-disciplinary areas including motion signal processing, machine learning (ML), sign language modeling, and real-time ML with dynamic device/edge partitioning, to develop new technology for automatic Sign Language Recognition (SLR) and translation to spoken language that enables more seamless communication between deaf and hearing people. The technology will incorporate wearable devices (such as a smartwatch, smart ring, and earphones) that are gaining in popularity, and will have broad impact through its introduction in deaf communities along with a sign language equivalent of voice assistants such as Amazon Alexa. The project will establish a pipeline of collaboration with deaf students, as well as courses based on SLR technology that will be disseminated through MOOC platforms such as Coursera. Additional impact will derive from workshops on wearable computing that will be conducted at the K-12 level, and a "sign-to-speech" library that will be publicly released for extensibility of the new technology to multiple sign languages. To achieve its goals this research will include three thrusts: Development of ML models with efficient training that can perform accurate SLR by fusing multimodal input data from wearable devices that capture body motion and facial expressions; Implementation of efficient ML models by means of optimal partitioning between end-device and edge resources to achieve the best tradeoff in real time performance and SLR accuracy; Design of systematic user studies with fluent sign language users both for generating training data for ML models as well as for validation of accuracy, usability, and acceptability of the technology within the deaf community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目将推进跨学科领域的最新技术，包括运动信号处理，机器学习（ML），手语建模和具有动态设备/边缘分区的实时ML，以开发自动手语识别（SLR）和翻译为口语的新技术，从而实现聋人和听力正常人之间的无缝沟通。该技术将结合越来越受欢迎的可穿戴设备（如智能手表，智能戒指和耳机），并将通过在聋人社区的引入产生广泛的影响，沿着手语相当于语音助手，如亚马逊Alexa。该项目将建立与聋人学生的合作渠道，以及基于SLR技术的课程，这些课程将通过Coursera等MOOC平台传播。将在K-12一级举办的可穿戴计算讲习班以及将公开发布的“手语转换”图书馆将产生额外的影响，以便将新技术扩展到多种手语。为了实现其目标，这项研究将包括三个重点：开发具有有效训练的ML模型，通过融合来自捕捉身体运动和面部表情的可穿戴设备的多模态输入数据来执行精确的SLR;通过在终端设备和边缘资源之间进行最佳划分来实现有效的ML模型，以实现真实的时间性能和SLR精度的最佳权衡;使用流利的手语用户设计系统的用户研究，用于生成ML模型的训练数据以及验证准确性，可用性，该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的评估来支持。影响审查标准。

项目成果

One Ring to Rule Them All: An Open Source Smartring Platform for Finger Motion Analytics and Healthcare Applications

• DOI: 10.1145/3576842.3582382
• 发表时间: 2023-05
• 期刊: Proceedings of the 8th ACM/IEEE Conference on Internet of Things Design and Implementation
• 作者: Hao Zhou;Taiting Lu;Yilin Liu;Shijia Zhang;Runze Liu;Mahanth K. Gowda

SignQuery: A Natural User Interface and Search Engine for Sign Languages with Wearable Sensors

• DOI: 10.1145/3570361.3613286
• 发表时间: 2023-10
• 期刊: Proceedings of the 29th Annual International Conference on Mobile Computing and Networking
• 作者: Hao Zhou;Taiting Lu;Kristina Mckinnie;Joseph Palagano;Kenneth DeHaan;Mahanth K. Gowda

Leveraging the Properties of mmWave Signals for 3D Finger Motion Tracking for Interactive IoT Applications

• DOI: 10.1145/3570613
• 发表时间: 2022-12
• 期刊: Proceedings of the ACM on Measurement and Analysis of Computing Systems
• 作者: Yilin Liu;Shijia Zhang;Mahanth K. Gowda;Srihari Nelakuditi

Let's Grab a Drink: Teacher-Student Learning for Fluid Intake Monitoring using Smart Earphones

• DOI: 10.1109/iotdi54339.2022.00014
• 发表时间: 2022-05
• 期刊: 2022 IEEE/ACM Seventh International Conference on Internet-of-Things Design and Implementation (IoTDI)
• 作者: Shijia Zhang;Yilin Liu;Mahanth K. Gowda