SBIR Phase I: Video-to-speech software application to provide real-time, noninvasive, natural voice restoration for voiceless individuals

这项小型企业创新研究（SBIR）第一阶段项目的更广泛影响旨在使100万美国人能够通过喉部或口腔疾病或损伤（失声）丧失说话能力。无法与他人流畅地交流会产生严重的后果。无声的人在医疗环境中遭受可预防的不良事件的可能性是说话患者的三倍，这可能导致健康问题，甚至危及生命。通过患者和临床医生之间的充分沟通，可以避免高达50%的不良事件。提出的解决方案是一个视频到语音的软件应用程序，提供无障碍的人与实时通信援助，特别是面向医疗设置。这项技术每年可以帮助预防数十万起不良健康事件（每年花费68亿美元），对无病人群和整个医疗保健系统都有好处。这项创新可以通过提供无需培训的实时翻译来改善语音恢复，并允许在眼睛对眼睛（人类交流的重要组成部分）的同时表达复杂的信息。此外，该技术不需要侵入性安装，也不需要复杂的设备，易于使用，并且具有边际维护要求。这个小型企业创新研究（SBIR）第一阶段项目旨在解决在尝试自动化唇读时克服视位模糊性的智力挑战。视素（说话时的手势）和音素（这些手势产生的声音）并不一一对应。这使得基于视觉信息准确预测预期语音具有挑战性。以前的研究人员在解释视位时未能达到可接受的准确度，而其他工具只能处理几十个必须根据预定义的固定规则结构化的单词，这些规则是不切实际的。这项工作的主要目标是开发卷积神经网络和递归神经网络传感器的组合，能够准确区分视位，并为无障碍人士提供实时，可靠的语音帮助。项目目标包括：（1）使用公开可用的语音视频对算法进行预训练以检测音素，（2）针对医疗保健相关词汇优化音素训练算法，以及（3）针对实时语音对唇读算法进行阿尔法测试。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估而被认为值得支持。

The broader impact of this Small Business Innovation Research (SBIR) Phase I project seeks to enable one million Americans that suffer with the loss of ability to speak through disease of or damage to the larynx or mouth (aphonia). The inability to fluently communicate with other people has severe consequences. Voiceless individuals are three times more likely to suffer a preventable adverse event in medical settings than speaking patients, and this can lead to health problems and even life-threatening situations. Up to 50% of these adverse events could be avoided with adequate communication between patients and clinicians. The proposed solution is a video-to-speech software application that provides voiceless people with real-time communication assistance, especially geared towards medical settings. The technology could help prevent hundreds of thousands of adverse health events each year (costing $6.8 billion annually), with benefits for the voiceless population and the healthcare system in general. The innovation may improve voice restoration by providing real-time translation with no training needed and allowing complex messages to be expressed while looking eye-to-eye (an important part of human communication). Moreover, the technology does not require invasive installations nor complex equipment, is readily accessible, and has maintenance requirements that are marginal.This Small Business Innovation Research (SBIR) Phase I project aims to address the intellectual challenge of overcoming the ambiguity of visemes when trying to automate lip-reading. Visemes (the gestures made when talking) and phonemes (the sounds produced with these gestures) do not share a one-to-one correspondence. This makes accurately predicting the intended speech based on visual information challenging. Previous researchers have failed to reach acceptable accuracy levels in the interpretation of visemes, while other tools only work with a few dozen words that must be structured according to pre-defined, fixed rules that are impractical. The main goal of this effort is to develop a combination of convolutional neural networks and recurrent neural network transducers that is capable of accurately differentiating visemes and permits real-time, reliable voice assistance for voiceless people. Project objectives include: (1) pre-training an algorithm to detect phonemes using publicly available speech video, (2) optimizing the phoneme-trained algorithm against healthcare relevant vocabulary, and (3) alpha-testing of the lip-reading algorithm against real-time speech.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.