EAGER: Lip Reading by Unobtrusive Multimodal Sensors and Machine Learning Algorithms

该项目旨在建立一个不显眼的系统，为肌萎缩性侧索硬化症（ALS，也称为Lou Gehrig's病）患者和有语言和听力障碍的人提供唇读服务。虽然有丰富的文献关于唇读，体积大，突兀，和/或固定这些解决方案阻碍了他们在日常实践中的应用，特别是对患者的神经肌肉疾病。迫切需要开发新的唇读技术，以提高ALS患者与亲人和医疗保健提供者的沟通能力。提出的系统可以大大改进现有的解决方案，用于跟踪和解释面部运动，更广泛地说，身体运动，如手指运动和身体手势。从不显眼的传感器收集多模态运动模式并应用机器学习（ML）来解释获取的数据的能力将极大地促进运动相关疾病（如中风和帕金森病）的诊断、治疗和康复。此外，这项工作为通过面部/身体手势实现的非语言交流界面的发展铺平了道路，并为康复、机器人和人机界面开辟了新的途径。本项目为学生参与跨学科研究提供了极好的机会。部分研究将被整合到PI的课程和顶点设计项目中。pi致力于通过当地少数民族组织和石溪大学的垂直整合计划开展外展活动并增加多样性。这个项目的首要目标是为ALS患者建立一个不显眼的硬件软件平台，可以捕捉与语言相关的嘴唇手势，并解码语言的嘴唇运动。首先，将设计一种类似皮肤的多模态应变和肌电图（EMG）传感系统，以跟踪与嘴唇运动相关的皮肤变形和肌肉活动。将引入自组装结构，使传感器超薄、透气和半透明。其次，将检测到的唇语信号转换为相应的口语的可行性进行论证。现代机器学习方法，特别是集成高斯过程（GPs）将被用于语音识别。在该方案中，每个GP作为分类器，并利用贝叶斯框架内的方法将所有GP的结果融合在一起进行最终决策。本研究的潜在贡献包括：1)通过可扩展的自组装工艺设计具有高灵敏度和良好皮肤相容性的类皮肤应变和肌电传感器。2)整合多模态传感器，对与言语相关的唇部运动进行全面的体内量化。3)开发将嘴唇动作精确转换为语音的ML算法。4)为开发一个真正自然而不显眼的唇读硬件软件系统奠定了基础。我们提出的工作可以通过直观和不显眼的唇读技术填补现有解决方案的空白。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The project aims to build an unobtrusive system to enable lip reading for patients with Amyotrophic Lateral Sclerosis (ALS, also known as Lou Gehrig's diseases) and individuals with speech and hearing disorders. Although there is rich literature on lip reading, the bulkiness, obtrusiveness, and/or immobility of these solutions impedes their applications in daily practice, especially for patients with neuromuscular disorders. There is an urgent need to develop novel lip-reading technologies to improve the communication capabilities of ALS patients with loved ones and healthcare providers. The proposed system can considerably improve on existing solutions for tracking and interpreting facial movements and more broadly, body movements, such as finger motions and body gestures. The ability to gather multimodal motion patterns from unobtrusive sensors and apply machine learning (ML) to interpret the acquired data would greatly facilitate diagnosis, treatment, and rehabilitation of motion-related disorders, such as stroke and Parkinson's disease. In addition, this work paves the way for the development of nonverbal communication interfaces enabled by facial/body gestures and opens new avenues for rehabilitation, robotics, and human-machine interfaces. This project presents an excellent opportunity for students to participate in cross-disciplinary research. Part of the research will be integrated into the PI's courses and capstone design projects. The PIs are committed to outreach activities and increasing the diversity through local minority organizations and the Vertically Integrated Program at Stony Brook University. The overarching goal of this project is to build an unobtrusive hardware-software platform for ALS patients that can capture speech-relevant lip gestures and decode lip movements for speech. First, a skin-like multimodal strain and electromyography (EMG) sensing system will be designed to track both skin deformations and muscle activities associated with lip movements. Self-assembled structures will be introduced to render the sensors ultrathin, breathable, and semi-transparent. Second, the feasibility of converting the sensed lip signals to corresponding spoken words will be demonstrated. Modern ML methods, and in particular, ensemble Gaussian processes (GPs) will be exploited for speech recognition. In the proposed scheme, each GP serves as a classifier and the final decision is made by fusing the results of all the GPs by making use of methods within the Bayesian framework. The potential contributions of the proposed work include: 1) Design of skin-like strain and EMG sensors with high sensitivity and good skin compatibility through a scalable self-assembly process. 2) Integration of multimodal sensors for comprehensive in-vivo quantification of lip movements associated with speech. 3) Development of ML algorithms that precisely convert lip movements to speech. 4) Laying the grounds for developing a truly natural and unobtrusive hardware-software system for lip reading. Our proposed work can fill the gaps in the existing solutions by an intuitive and unobtrusive technology for lip reading.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.