EAGER: Lip Reading by Unobtrusive Multimodal Sensors and Machine Learning Algorithms

EAGER：通过不显眼的多模态传感器和机器学习算法进行唇读

• 批准号: 2129673
• 负责人: Shanshan Yao
• 金额: $ 14.99万
• 依托单位: SUNY at Stony Brook
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-08-15 至 2023-01-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2129673&HistoricalAwards=false
• 关键词: EAGER; Lip; Reading; Unobtrusive; Multimodal

The project aims to build an unobtrusive system to enable lip reading for patients with Amyotrophic Lateral Sclerosis (ALS, also known as Lou Gehrig's diseases) and individuals with speech and hearing disorders. Although there is rich literature on lip reading, the bulkiness, obtrusiveness, and/or immobility of these solutions impedes their applications in daily practice, especially for patients with neuromuscular disorders. There is an urgent need to develop novel lip-reading technologies to improve the communication capabilities of ALS patients with loved ones and healthcare providers. The proposed system can considerably improve on existing solutions for tracking and interpreting facial movements and more broadly, body movements, such as finger motions and body gestures. The ability to gather multimodal motion patterns from unobtrusive sensors and apply machine learning (ML) to interpret the acquired data would greatly facilitate diagnosis, treatment, and rehabilitation of motion-related disorders, such as stroke and Parkinson's disease. In addition, this work paves the way for the development of nonverbal communication interfaces enabled by facial/body gestures and opens new avenues for rehabilitation, robotics, and human-machine interfaces. This project presents an excellent opportunity for students to participate in cross-disciplinary research. Part of the research will be integrated into the PI's courses and capstone design projects. The PIs are committed to outreach activities and increasing the diversity through local minority organizations and the Vertically Integrated Program at Stony Brook University. The overarching goal of this project is to build an unobtrusive hardware-software platform for ALS patients that can capture speech-relevant lip gestures and decode lip movements for speech. First, a skin-like multimodal strain and electromyography (EMG) sensing system will be designed to track both skin deformations and muscle activities associated with lip movements. Self-assembled structures will be introduced to render the sensors ultrathin, breathable, and semi-transparent. Second, the feasibility of converting the sensed lip signals to corresponding spoken words will be demonstrated. Modern ML methods, and in particular, ensemble Gaussian processes (GPs) will be exploited for speech recognition. In the proposed scheme, each GP serves as a classifier and the final decision is made by fusing the results of all the GPs by making use of methods within the Bayesian framework. The potential contributions of the proposed work include: 1) Design of skin-like strain and EMG sensors with high sensitivity and good skin compatibility through a scalable self-assembly process. 2) Integration of multimodal sensors for comprehensive in-vivo quantification of lip movements associated with speech. 3) Development of ML algorithms that precisely convert lip movements to speech. 4) Laying the grounds for developing a truly natural and unobtrusive hardware-software system for lip reading. Our proposed work can fill the gaps in the existing solutions by an intuitive and unobtrusive technology for lip reading.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目旨在建立一个不显眼的系统，使唇阅读的患者肌萎缩侧索硬化症（ALS，也称为卢格里克病）和个人的语言和听力障碍。虽然有丰富的文献唇阅读，笨重，突兀，和/或固定的这些解决方案阻碍了他们的应用在日常实践中，特别是对患者的神经肌肉疾病。迫切需要开发新的唇读技术，以提高ALS患者与亲人和医疗保健提供者的沟通能力。所提出的系统可以大大改善现有的解决方案，用于跟踪和解释面部运动，更广泛地说，身体运动，如手指运动和身体姿势。从不显眼的传感器收集多模态运动模式并应用机器学习（ML）来解释所获取的数据的能力将极大地促进中风和帕金森病等运动相关疾病的诊断、治疗和康复。此外，这项工作为通过面部/身体手势实现的非语言通信接口的开发铺平了道路，并为康复，机器人和人机接口开辟了新的途径。该项目为学生提供了一个参与跨学科研究的绝佳机会。部分研究将被整合到PI的课程和顶点设计项目中。PI致力于通过当地少数民族组织和斯托尼布鲁克大学的纵向综合方案开展外联活动和增加多样性。该项目的总体目标是为ALS患者构建一个不引人注目的硬件软件平台，该平台可以捕获与语音相关的嘴唇手势并解码语音的嘴唇运动。首先，一个类似皮肤的多模态应变和肌电图（EMG）传感系统将被设计为跟踪皮肤变形和肌肉活动与嘴唇运动。将引入自组装结构，使传感器超薄、透气和半透明。其次，将证明将感测到的唇信号转换为对应的口语单词的可行性。现代ML方法，特别是集成高斯过程（GP）将用于语音识别。在所提出的方案中，每个GP作为一个分类器和最终的决定是通过融合所有的GP的结果，利用贝叶斯框架内的方法。本论文的潜在贡献包括：1）通过可扩展的自组装工艺，设计具有高灵敏度和良好皮肤相容性的类皮肤应变和肌电信号传感器。2)多模态传感器的集成，用于与语音相关的嘴唇运动的全面体内量化。3)开发ML算法，精确地将嘴唇运动转换为语音。4)为开发一个真正自然和不引人注目的唇阅读硬件软件系统奠定了基础。我们提出的工作可以填补现有解决方案的空白，通过一个直观和不显眼的技术唇阅读。这个奖项反映了NSF的法定使命，并已被认为是值得通过评估使用基金会的智力价值和更广泛的影响审查标准的支持。

项目成果

Decoding silent speech commands from articulatory movements through soft magnetic skin and machine learning

通过软磁皮肤和机器学习从发音运动解码无声语音命令

• DOI: 10.1039/d3mh01062g
• 发表时间: 2023
• 期刊: Materials Horizons
• 影响因子: 13.3
• 作者: Dong, Penghao;Li, Yizong;Chen, Si;Grafstein, Justin T.;Khan, Irfaan;Yao, Shanshan
• 通讯作者: Yao, Shanshan

A multi-tasking model of speaker-keyword classification for keeping human in the loop of drone-assisted inspection

说话者关键词分类的多任务模型，使人类能够参与无人机辅助检查的循环

• DOI: 10.1016/j.engappai.2022.105597
• 发表时间: 2023
• 期刊: Engineering Applications of Artificial Intelligence
• 影响因子: 8
• 作者: Li, Yu;Parsan, Anisha;Wang, Bill;Dong, Penghao;Yao, Shanshan;Qin, Ruwen
• 通讯作者: Qin, Ruwen