EAGER: Lip Reading by Unobtrusive Multimodal Sensors and Machine Learning Algorithms

该项目旨在建立一个非侵入性的系统，为肌萎缩侧索硬化症(ALS，也称为Lou Gehrig病)患者以及患有语言和听力障碍的个人提供唇读功能。虽然有大量关于唇读的文献，但这些解决方案的笨重、突出和/或固定阻碍了它们在日常实践中的应用，特别是对神经肌肉疾病患者。迫切需要开发新的唇读技术，以提高ALS患者与亲人和医疗保健提供者的沟通能力。拟议的系统可以大大改进现有的跟踪和解释面部动作以及更广泛地说，身体动作，如手指动作和身体手势的解决方案。从不显眼的传感器收集多模式运动模式并应用机器学习(ML)来解释所获得的数据的能力将极大地促进运动相关疾病的诊断、治疗和康复，如中风和帕金森病。此外，这项工作为通过面部/身体手势实现非语言交流界面的开发铺平了道路，并为康复、机器人和人机界面开辟了新的途径。这个项目为学生提供了一个参与跨学科研究的绝佳机会。部分研究将整合到PI的课程和顶峰设计项目中。PIs致力于通过当地少数族裔组织和石溪大学的垂直整合计划开展外联活动，增加多样性。该项目的首要目标是为肌萎缩侧索硬化症患者建立一个不引人注目的硬件-软件平台，该平台可以捕获与语音相关的嘴唇手势，并解码嘴唇动作以供语音使用。首先，将设计一种类似皮肤的多模式应变和肌电(EMG)传感系统，以跟踪与嘴唇运动相关的皮肤变形和肌肉活动。将引入自组装结构，使传感器超薄、透气和半透明。其次，将论证将感测到的嘴唇信号转换为对应的口语单词的可行性。现代最大似然方法，特别是集成高斯过程(GP)将被用于语音识别。在该方案中，每个GP作为一个分类器，通过使用贝叶斯框架内的方法融合所有GP的结果来做出最终决策。这项工作的潜在贡献包括：1)通过可扩展的自组装过程设计出具有高灵敏度和良好皮肤兼容性的类皮肤应变和肌电传感器。2)集成多模式传感器，用于在活体内全面量化与语音相关的嘴唇运动。3)开发可将嘴唇动作精确转换为语音的ML算法。4)为开发一个真正自然和不引人注目的唇读硬件软件系统奠定了基础。我们建议的工作可以通过一种直观和不引人注目的唇读技术来填补现有解决方案的空白。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The project aims to build an unobtrusive system to enable lip reading for patients with Amyotrophic Lateral Sclerosis (ALS, also known as Lou Gehrig's diseases) and individuals with speech and hearing disorders. Although there is rich literature on lip reading, the bulkiness, obtrusiveness, and/or immobility of these solutions impedes their applications in daily practice, especially for patients with neuromuscular disorders. There is an urgent need to develop novel lip-reading technologies to improve the communication capabilities of ALS patients with loved ones and healthcare providers. The proposed system can considerably improve on existing solutions for tracking and interpreting facial movements and more broadly, body movements, such as finger motions and body gestures. The ability to gather multimodal motion patterns from unobtrusive sensors and apply machine learning (ML) to interpret the acquired data would greatly facilitate diagnosis, treatment, and rehabilitation of motion-related disorders, such as stroke and Parkinson's disease. In addition, this work paves the way for the development of nonverbal communication interfaces enabled by facial/body gestures and opens new avenues for rehabilitation, robotics, and human-machine interfaces. This project presents an excellent opportunity for students to participate in cross-disciplinary research. Part of the research will be integrated into the PI's courses and capstone design projects. The PIs are committed to outreach activities and increasing the diversity through local minority organizations and the Vertically Integrated Program at Stony Brook University. The overarching goal of this project is to build an unobtrusive hardware-software platform for ALS patients that can capture speech-relevant lip gestures and decode lip movements for speech. First, a skin-like multimodal strain and electromyography (EMG) sensing system will be designed to track both skin deformations and muscle activities associated with lip movements. Self-assembled structures will be introduced to render the sensors ultrathin, breathable, and semi-transparent. Second, the feasibility of converting the sensed lip signals to corresponding spoken words will be demonstrated. Modern ML methods, and in particular, ensemble Gaussian processes (GPs) will be exploited for speech recognition. In the proposed scheme, each GP serves as a classifier and the final decision is made by fusing the results of all the GPs by making use of methods within the Bayesian framework. The potential contributions of the proposed work include: 1) Design of skin-like strain and EMG sensors with high sensitivity and good skin compatibility through a scalable self-assembly process. 2) Integration of multimodal sensors for comprehensive in-vivo quantification of lip movements associated with speech. 3) Development of ML algorithms that precisely convert lip movements to speech. 4) Laying the grounds for developing a truly natural and unobtrusive hardware-software system for lip reading. Our proposed work can fill the gaps in the existing solutions by an intuitive and unobtrusive technology for lip reading.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.