CAREER: Integrating perceptual models of auditory importance into deep learning-based noise-robust speech recognition

听觉是人类互动的核心，但听觉过程不容易观察到。这个项目的目标是训练模型来识别对人类听众正确识别语音很重要的部分，并使用这些模型的预测，通过关注这些区域，使自动语音识别（ASR）系统具有更强的噪声鲁棒性。识别话语中重要区域的能力可以大大提高我们对健康听力和受损听力的理解。自动语音识别的改进将对2.6亿使用智能手机的美国人以及价值1000亿美元的自动语音识别行业产生更广泛的影响。这个项目的教育部分利用了语音、语言、音频和音乐处理的例子，以吸引和留住布鲁克林学院的入门编程课程的学生，为不同的学生群体提供服务，并在附属高中项目中做出类似的努力。该团队的初步结果表明，通过测量在许多不同的噪声混合中给定话语的可理解性，话语的某些区域比其他区域更重要或更有用。本项目从三个方面扩展了这些初步结果。首先，它使用该团队现有的缓慢但准确的随机“气泡噪声”技术来衡量ASR听觉的重要性，将不同的ASR变体相互比较，并与人类听众进行比较。其次，它使用一种称为气泡合作网络（BCN）的新架构训练一个模型来预测干净语音的ASR听觉重要性，该架构允许识别器与BCN共同训练以提高性能。第三，它将学习到的重要性预测器应用于人类听众，并使用这种人类适应的重要性预测器进一步完善ASR模型。这些任务应该允许使用话语水平的人类反应来直接提高自动语音识别的噪声鲁棒性。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Hearing is central to human interaction, but the hearing process is not easily observed. The objective of this project is to train models to identify portions of speech utterances that are important to their being correctly identified by human listeners, and to use predictions from these models to make automatic speech recognition (ASR) systems more noise robust by focusing on those regions. The ability to identify important regions of an utterance could significantly advance our understanding of healthy and impaired hearing. Improvements in automatic speech recognition would have broader impacts on the 260 million Americans who use smart phones and the $100 billion ASR industry. The educational portion of this project utilizes examples from speech, language, audio, and music processing to attract and retain students in Brooklyn College's introductory programming course serving a diverse student body along with similar efforts at affiliated high school programs.The team's preliminary results have shown that that some regions of an utterance are more important or useful than others in identifying it by measuring the intelligibility of a given utterance in many different noisy mixtures. This project expands upon these preliminary results in three ways. First it measures ASR auditory importance using the team's existing slow but accurate technique involving random "bubble noise", comparing different ASR variants to each other and to human listeners. Second, it trains a model to predict ASR auditory importance from clean speech using a novel architecture called the bubble cooperative network (BCN) that allows the recognizer to be trained jointly with the BCN to improve performance. Third, it adapts the learned importance predictor to human listeners and uses this human-adapted importance predictor to further refine the ASR models. These tasks should permit the use of utterance-level human responses to directly improve the noise robustness of automatic speech recognition.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.