CAREER: Integrating perceptual models of auditory importance into deep learning-based noise-robust speech recognition

听力是人类互动的中心，但听力过程不易观察到。这个项目的目标是训练模型来识别对人类听众正确识别语音发声非常重要的部分，并使用来自这些模型的预测来通过关注这些区域来使自动语音识别(ASR)系统具有更强的噪声鲁棒性。识别话语中重要区域的能力可以显著提高我们对听力健康和听力受损的理解。自动语音识别的改进将对使用智能手机的2.6亿美国人和价值1000亿美元的ASR行业产生更广泛的影响。这个项目的教育部分利用语音、语言、音频和音乐处理的例子来吸引和留住布鲁克林学院为不同学生群体服务的入门编程课程的学生，以及附属高中的类似努力。该团队的初步结果表明，通过测量给定话语在许多不同噪声混合中的可理解性，某个话语的某些区域比其他区域更重要或更有用。该项目从三个方面对这些初步结果进行了扩展。首先，它使用团队现有的缓慢但准确的技术来测量ASR的听觉重要性，该技术涉及随机的“气泡噪音”，将不同的ASR变体彼此比较，并与人类听众进行比较。其次，它使用一种称为气泡合作网络(BCN)的新型体系结构来训练一个模型，以从干净的语音中预测ASR的听觉重要性，该体系结构允许识别器与BCN联合训练以提高性能。第三，它使习得的重要性预测器适应人类听众，并使用这种人类适应的重要性预测器来进一步改进ASR模型。这些任务应该允许使用发声级别的人类反应来直接提高自动语音识别的噪声稳健性。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Hearing is central to human interaction, but the hearing process is not easily observed. The objective of this project is to train models to identify portions of speech utterances that are important to their being correctly identified by human listeners, and to use predictions from these models to make automatic speech recognition (ASR) systems more noise robust by focusing on those regions. The ability to identify important regions of an utterance could significantly advance our understanding of healthy and impaired hearing. Improvements in automatic speech recognition would have broader impacts on the 260 million Americans who use smart phones and the $100 billion ASR industry. The educational portion of this project utilizes examples from speech, language, audio, and music processing to attract and retain students in Brooklyn College's introductory programming course serving a diverse student body along with similar efforts at affiliated high school programs.The team's preliminary results have shown that that some regions of an utterance are more important or useful than others in identifying it by measuring the intelligibility of a given utterance in many different noisy mixtures. This project expands upon these preliminary results in three ways. First it measures ASR auditory importance using the team's existing slow but accurate technique involving random "bubble noise", comparing different ASR variants to each other and to human listeners. Second, it trains a model to predict ASR auditory importance from clean speech using a novel architecture called the bubble cooperative network (BCN) that allows the recognizer to be trained jointly with the BCN to improve performance. Third, it adapts the learned importance predictor to human listeners and uses this human-adapted importance predictor to further refine the ASR models. These tasks should permit the use of utterance-level human responses to directly improve the noise robustness of automatic speech recognition.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.