Collaborative Research: RI: Medium: Flexible Deep Speech Synthesis through Gestural Modeling

从汽车、手机到数字家庭助理，基于语音的互动已经成为无处不在的常态。随着基于语音的机器交互变得越来越普遍，人们对这些系统的类人性能和个性的需求和期望也越来越高。对于机器来说，在一个阳光明媚的日子或即将来临的飓风中，以适当的方式提供有关天气的响应是很重要的。机器需要能够根据使用环境做出同情或强调的反应。关键的是，当机器出现故障时，它们应该以人类可以理解的方式出现故障，这样就不会出现技术带来的意外后果。该项目旨在创造更自然和灵活的语音合成技术，该技术的灵感来自于人类的语音生产策略和机制。该项目将语音生成科学和当前最先进的工程语音系统结合在一起，旨在赋予语音技术可解释性、自然性和灵活性。这个项目有可能影响所有使用语音输出的系统，如自动辅导、交互式语音响应、商业和军事环境中的语音翻译、数字助理、机器人和康复医疗应用，如脑机接口。当前的语音合成技术主要集中在端到端系统上，避免了对语音信号内部结构的显式建模。因此，这样的系统可能有良好的结果，但不允许任何泛化超出其记录的数据库。这个项目的重点是将人类语音生产的各个方面整合到计算机语音合成中。利用数据驱动技术和声道成像数据集，该项目旨在发现和模拟由发音音韵学描述的语音信号的组成方面。将开发新的基于深度学习的方法，用于在合成分析框架内联合优化各种语音表示，如声学、音系和生理数据。将开发新的策略，将基础表示纳入文本到语音的训练，并在灵活语音合成的一系列应用中进行评估。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Voice based interactions have become the norm everywhere from cars, to mobile phones to digital home assistants. As speech based machine interaction becomes more pervasive, there is increased demand and expectation of human-like performance and personality from these systems. It is important for the machine to deliver responses about the weather on a pleasant sunny day or an impending hurricane in an appropriate manner. Machines need to be able to respond sympathetically or emphatically depending on the context of their use. Critically, when machines fail, they should do so in human understandable ways, so that there are no unintended consequences of technology. This project aims to create more natural and flexible speech synthesis technology that is inspired by human strategies and mechanisms for speech production. Bringing together the science of speech production and current state-of-the-art engineering speech systems, this project aims to impart explainability, naturalness and flexibility to speech technologies. This project has the potential to impact all systems that use speech output like automated tutoring, interactive voice response, speech translation in commercial and military settings, digital assistants, robotics and rehabilitative healthcare applications like Brain-Computer Interfaces. Current speech synthesis techniques are focused on end-to-end systems, avoiding explicit modeling of internal structure of the speech signal. Consequently, such systems may have good results but fail to allow any generalization beyond their recorded databases. This project concentrates on incorporating aspects of human speech production into computer speech synthesis. Using data-driven techniques and vocal tract imaging datasets, the project aims to discover and model compositional aspects of the speech signal as described by Articulatory Phonology. Novel deep-learning based approaches will be developed for joint optimization of diverse speech representations such as acoustic, phonological and physiological data within an analysis-by-synthesis framework. New strategies will be developed for incorporating grounded representations into text-to-speech training and evaluated in a range of applications in flexible speech synthesis.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.