CAREER: Sign-to-Speech: An Edge-IoT Platform and Software Library for Real Time Sign Language Recognition

该项目将推动包括运动信号处理、机器学习(ML)、手语建模和具有动态设备/边缘划分的实时ML在内的跨学科领域的最先进技术，以开发自动手语识别(SLR)和翻译成口语的新技术，使聋人和听力正常的人之间能够更无缝地交流。这项技术将结合越来越受欢迎的可穿戴设备(如智能手表、智能戒指和耳机)，并将在聋人社区引入，以及亚马逊Alexa等相当于语音助手的手语，从而产生广泛影响。该项目将建立一个与聋人学生合作的渠道，以及将通过Coursera等MOOC平台传播的基于单反技术的课程。其他影响将来自将在K-12级别进行的可穿戴计算研讨会，以及将公开发布的“手语到语音”库，以将新技术扩展到多种手语。为了实现其目标，这项研究将包括三个方面：开发具有高效训练的ML模型，该模型可以通过融合来自可穿戴设备的捕捉身体运动和面部表情的多模式输入数据来执行准确的单反；通过在终端设备和边缘资源之间的优化划分来实现高效的ML模型，以实现实时性能和单反精度的最佳折衷；与流利的手语用户一起设计系统的用户研究，为ML模型生成训练数据，以及验证该技术在聋人社区中的准确性、可用性和可接受性。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project will advance the state-of-the-art in cross-disciplinary areas including motion signal processing, machine learning (ML), sign language modeling, and real-time ML with dynamic device/edge partitioning, to develop new technology for automatic Sign Language Recognition (SLR) and translation to spoken language that enables more seamless communication between deaf and hearing people. The technology will incorporate wearable devices (such as a smartwatch, smart ring, and earphones) that are gaining in popularity, and will have broad impact through its introduction in deaf communities along with a sign language equivalent of voice assistants such as Amazon Alexa. The project will establish a pipeline of collaboration with deaf students, as well as courses based on SLR technology that will be disseminated through MOOC platforms such as Coursera. Additional impact will derive from workshops on wearable computing that will be conducted at the K-12 level, and a "sign-to-speech" library that will be publicly released for extensibility of the new technology to multiple sign languages. To achieve its goals this research will include three thrusts: Development of ML models with efficient training that can perform accurate SLR by fusing multimodal input data from wearable devices that capture body motion and facial expressions; Implementation of efficient ML models by means of optimal partitioning between end-device and edge resources to achieve the best tradeoff in real time performance and SLR accuracy; Design of systematic user studies with fluent sign language users both for generating training data for ML models as well as for validation of accuracy, usability, and acceptability of the technology within the deaf community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.