SBIR Phase I: Video-to-speech software application to provide real-time, noninvasive, natural voice restoration for voiceless individuals

这个小企业创新研究(SBIR)第一阶段项目的更广泛影响旨在帮助100万因喉咙或口腔疾病或损伤(失音)而丧失说话能力的美国人。不能与他人流畅地沟通会带来严重的后果。失声的人在医疗环境中遭受可预防的不良事件的可能性是说话的患者的三倍，这可能会导致健康问题，甚至危及生命的情况。如果患者和临床医生之间有足够的沟通，这些不良事件中高达50%可以避免。建议的解决方案是一种视频到语音的软件应用程序，它为无声的人提供实时通信帮助，特别是针对医疗设置。这项技术可以帮助防止每年数十万次不良健康事件(每年花费68亿美元)，并使无声人口和整个医疗体系受益。这项创新可能会改善语音恢复，方法是提供无需培训的实时翻译，并允许在眼睛对眼睛的情况下表达复杂的信息(这是人类交流的重要组成部分)。此外，该技术不需要侵入性安装或复杂的设备，很容易获得，并且维护要求很低。这个小型企业创新研究(SBIR)第一阶段项目旨在解决在尝试自动化唇读时克服视位模棱两可的智力挑战。视位(说话时的手势)和音位(用这些手势发出的声音)并不是一一对应的。这使得基于视觉信息准确预测预期语音具有挑战性。之前的研究人员在解释视位时未能达到可以接受的准确度水平，而其他工具只能处理几十个词，这些词必须按照预先定义的固定规则进行结构，这是不切实际的。这项工作的主要目标是开发卷积神经网络和递归神经网络换能器的组合，能够准确区分视位，并为无声人提供实时、可靠的语音辅助。项目目标包括：(1)使用公开的语音视频预先训练检测音素的算法，(2)针对医疗保健相关词汇优化音素训练的算法，以及(3)针对实时语音对唇读算法进行阿尔法测试。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The broader impact of this Small Business Innovation Research (SBIR) Phase I project seeks to enable one million Americans that suffer with the loss of ability to speak through disease of or damage to the larynx or mouth (aphonia). The inability to fluently communicate with other people has severe consequences. Voiceless individuals are three times more likely to suffer a preventable adverse event in medical settings than speaking patients, and this can lead to health problems and even life-threatening situations. Up to 50% of these adverse events could be avoided with adequate communication between patients and clinicians. The proposed solution is a video-to-speech software application that provides voiceless people with real-time communication assistance, especially geared towards medical settings. The technology could help prevent hundreds of thousands of adverse health events each year (costing $6.8 billion annually), with benefits for the voiceless population and the healthcare system in general. The innovation may improve voice restoration by providing real-time translation with no training needed and allowing complex messages to be expressed while looking eye-to-eye (an important part of human communication). Moreover, the technology does not require invasive installations nor complex equipment, is readily accessible, and has maintenance requirements that are marginal.This Small Business Innovation Research (SBIR) Phase I project aims to address the intellectual challenge of overcoming the ambiguity of visemes when trying to automate lip-reading. Visemes (the gestures made when talking) and phonemes (the sounds produced with these gestures) do not share a one-to-one correspondence. This makes accurately predicting the intended speech based on visual information challenging. Previous researchers have failed to reach acceptable accuracy levels in the interpretation of visemes, while other tools only work with a few dozen words that must be structured according to pre-defined, fixed rules that are impractical. The main goal of this effort is to develop a combination of convolutional neural networks and recurrent neural network transducers that is capable of accurately differentiating visemes and permits real-time, reliable voice assistance for voiceless people. Project objectives include: (1) pre-training an algorithm to detect phonemes using publicly available speech video, (2) optimizing the phoneme-trained algorithm against healthcare relevant vocabulary, and (3) alpha-testing of the lip-reading algorithm against real-time speech.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.