SBIR Phase I: Video-to-speech software application to provide real-time, noninvasive, natural voice restoration for voiceless individuals

SBIR 第一阶段：视频转语音软件应用程序，为失声者提供实时、无创、自然的语音恢复

• 批准号: 2136629
• 负责人: Yi Han
• 金额: $ 25.6万
• 依托单位: LIRA, INC.
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-09-01 至 2023-11-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2136629&HistoricalAwards=false
• 关键词: SBIR; Phase; Video; speech; software

The broader impact of this Small Business Innovation Research (SBIR) Phase I project seeks to enable one million Americans that suffer with the loss of ability to speak through disease of or damage to the larynx or mouth (aphonia). The inability to fluently communicate with other people has severe consequences. Voiceless individuals are three times more likely to suffer a preventable adverse event in medical settings than speaking patients, and this can lead to health problems and even life-threatening situations. Up to 50% of these adverse events could be avoided with adequate communication between patients and clinicians. The proposed solution is a video-to-speech software application that provides voiceless people with real-time communication assistance, especially geared towards medical settings. The technology could help prevent hundreds of thousands of adverse health events each year (costing $6.8 billion annually), with benefits for the voiceless population and the healthcare system in general. The innovation may improve voice restoration by providing real-time translation with no training needed and allowing complex messages to be expressed while looking eye-to-eye (an important part of human communication). Moreover, the technology does not require invasive installations nor complex equipment, is readily accessible, and has maintenance requirements that are marginal.This Small Business Innovation Research (SBIR) Phase I project aims to address the intellectual challenge of overcoming the ambiguity of visemes when trying to automate lip-reading. Visemes (the gestures made when talking) and phonemes (the sounds produced with these gestures) do not share a one-to-one correspondence. This makes accurately predicting the intended speech based on visual information challenging. Previous researchers have failed to reach acceptable accuracy levels in the interpretation of visemes, while other tools only work with a few dozen words that must be structured according to pre-defined, fixed rules that are impractical. The main goal of this effort is to develop a combination of convolutional neural networks and recurrent neural network transducers that is capable of accurately differentiating visemes and permits real-time, reliable voice assistance for voiceless people. Project objectives include: (1) pre-training an algorithm to detect phonemes using publicly available speech video, (2) optimizing the phoneme-trained algorithm against healthcare relevant vocabulary, and (3) alpha-testing of the lip-reading algorithm against real-time speech.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

这项小企业创新研究（SBIR）第一阶段项目的更广泛影响是，寻求使100万因喉部或口腔疾病或损伤（失音症）而丧失说话能力的美国人能够说话。不能流利地与他人交流会带来严重的后果。在医疗环境中，无声者遭受可预防的不良事件的可能性是会说话的患者的三倍，这可能导致健康问题，甚至危及生命的情况。通过患者和临床医生之间的充分沟通，多达50%的不良事件可以避免。提出的解决方案是一个视频转语音软件应用程序，为无声者提供实时通信帮助，特别是针对医疗环境。这项技术每年可以帮助预防成千上万的不良健康事件（每年花费68亿美元），对无声人群和整个医疗系统都有好处。这项创新可能会通过提供无需训练的实时翻译来改善语音恢复，并允许在眼睛对视（人类交流的重要组成部分）的情况下表达复杂的信息。此外，该技术不需要侵入式安装，也不需要复杂的设备，易于获取，并且维护要求很少。这个小企业创新研究（SBIR）第一阶段项目旨在解决在尝试自动化唇读时克服假面模糊的智力挑战。语素（说话时做出的手势）和音素（用这些手势发出的声音）并不是一一对应的。这使得基于视觉信息的准确预测语音变得具有挑战性。先前的研究人员在解释字谜时未能达到可接受的精度水平，而其他工具只能处理几十个必须根据预先定义的固定规则构建的单词，这些规则是不切实际的。这项工作的主要目标是开发一种卷积神经网络和循环神经网络换能器的组合，该换能器能够准确地区分人脸，并为无声者提供实时、可靠的语音辅助。项目目标包括：(1)使用公开的语音视频预训练一种算法来检测音素；(2)针对医疗保健相关词汇优化音素训练算法；(3)针对实时语音对唇读算法进行alpha测试。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。