RI: Small: Concatenative Resynthesis for Very High Quality Speech Enhancement

环境噪声是助听器、移动电话和自动语音识别等语音技术用户面临的最大问题之一。当前用于源分离和语音增强的方法通常试图修改噪声信号以使其更像原始信号，从而导致目标语音和残留噪声中的失真。相比之下，这个项目使用了一种创新的方法，使用从噪声信号中提取的信息来驱动语音合成器，以创建全新的、高质量的、无噪音的原始句子版本。这种方法在噪音抑制和语音质量方面的改善，预计将对3600万听力受损的美国人和2亿使用智能手机的美国人产生重要的、更广泛的影响。该项目还被纳入一所多元化的城市大学的课程，并被纳入到附近高中的既定外联计划中，目的是鼓励代表性不足的群体的成员追求科学和工程职业。该项目旨在通过对串联语音合成器进行改造，使用基于新型深度神经网络(DNN)架构的单元选择功能，来产生高质量的语音再合成系统。初步结果表明，这种方法在小词汇量、依赖说话者的任务中效果很好，本项目通过三种方式将其扩展到大词汇量、依赖说话者的环境中。首先，它试图通过利用感知激励的输入特征、更灵活的训练信号和传统的语音增强来提高合成语音的可理解性。其次，它试图通过训练DNN将噪声和干净的语音嵌入到可以有效计算相似度的联合低维空间中来提高系统的可扩展性。第三，它试图通过结合基于声学、语音和语言兼容性的语音单元的顺序模型来提高合成语音的质量。语音合成模型在语音增强中的使用是对传统方法的偏离，并且有可能对增强的语音质量产生变革性的影响。

Environmental noise is one of the largest problem for users of voice technologies, such as hearing aids, mobile phones, and automatic speech recognition. Current approaches to source separation and speech enhancement typically attempt to modify the noisy signal in order to make it more like the original, leading to distortions in target speech and residual noise. In contrast, this project uses the innovative approach of driving a speech synthesizer using information extracted from the noisy signal to create a brand new, high quality, noise-free version of the original sentence. Improvements in noise suppression and speech quality from this approach are expected to have important broader impacts for both the 36 million Americans who are hearing impaired and the 200 million Americans who use smart phones. The project is also being incorporated into the curriculum in a diverse urban college and into established outreach programs to nearby high schools with the goal of encouraging members of under-represented groups to pursue careers in science and engineering.This project aims to produce a high quality speech resynthesis system by modifying a concatenative speech synthesizer to use a unit-selection function based on a novel deep neural network (DNN) architecture. Preliminary results have shown this approach to work well for a small-vocabulary, speaker-dependent task, and the current project expands this to the large-vocabulary, speaker-dependent setting in three ways. First, it seeks to improve the intelligibility of the synthesized speech by utilizing perceptually motivated input features, more flexible training signals, and traditional speech enhancement. Second, it seeks to improve the system's scalability by training DNNs to embed noisy and clean speech into a joint low-dimensional space in which similarity can be efficiently computed. And third, it seeks to improve the quality of the synthesized speech by incorporating sequential models of speech units based on acoustic, phonetic, and linguistic compatibility. The use of speech synthesis models in speech enhancement is a departure from traditional approaches and has the potential to make a transformative impact on the quality of enhanced speech.