Collaborative Research: Estimating Articulatory Constriction Place and Timing from Speech Acoustics

这个合作项目的重点是一种新的方法，使用语音记录来研究说话者的发音习惯-也就是说，说话者系统地协调他们的嘴唇，下巴，舌头，声门和软腭的发音运动，以产生单词和句子。 这些发音习惯在不同的个体之间、不同的语言之间以及同一语言的方言之间都有所不同，这就解释了外国口音、言语障碍和说话风格的许多方面。 鉴于这些习惯的以前的研究需要专门的设备来立即观察咬合运动，该项目的目的是开发和改进一种“语音反转”工具-也就是说，一种可以使用机器学习方法直接从声学语音信号中准确恢复发音运动的工具。 到目前为止，该项目团队开发的工具已经成功恢复了舌头和嘴唇的运动;目前的项目将该工具的功能扩展到包括鼻音（软腭）和发声（声门）。 扩展系统的训练和验证将继续使用新收集的语料库的声学和发音数据来自美国英语的发言者。 该语料库包括共同收集的音频、鼻音、发声和发音运动，将作为用于训练和评估完全训练的语音反转系统的能力的"地面实况"。 作为进一步的测试，我们将测试它对地面真实数据从语言的发音习惯模式已知不同于英语的扬声器。该项目的目标是开发和完善语音反转工具，"读取"语音的声学记录和"恢复"的细节的大小和发音运动的时间。 该项目旨在通过训练专门的神经网络模型来实现这一目标，以将声学信号的特征与分别采集的地面真实鼻与口流出信号和并发电声门图相关联。训练数据来自母语为英语的人;验证和测试泛化包括加拿大法语和俄语的发言者的作品。当成功验证后，所得到的语音反转工具将有助于识别影响语音运动组织的医疗问题，例如众所周知的发音障碍患者的口腔/喉部时序中断。此外，纳入对清晰度的估计也有助于跟踪由抑郁症和精神分裂症等医疗状况引起的变化。 更一般地，快速且容易地分析从单独的音频记录获得的发音运动的能力具有实质上改进自动语音识别（ASR）系统的潜力，并且有助于学者、法医科学家和临床专业人员在农村或资源不足地区的实地条件下研究社区的语音，该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This collaborative project focuses on a new approach for using speech recordings to study speaker pronunciation habits--that is, the way speakers systematically coordinate the articulatory movements of their lips, jaw, tongue, glottis and soft palate to produce words and sentences. These articulatory habits differ between individuals, and across languages and dialects of the same language, accounting for many aspects of foreign accent, speech disorders and speaking style. Whereas previous studies of these habits have required specialized equipment for the immediate observation of articulator movements, the aim of this project is to develop and improve a tool for "speech inversion"--that is, a tool that can accurately recover articulatory movements directly from the acoustic speech signal using machine learning methods. To date, the tool developed by the project team has successfully recovered movements of the tongue and lips; the current project extends the tool’s functionality to encompass nasality (soft palate) and voicing (glottis). Training and validation of the extended system will proceed using a newly collected corpus of acoustic and articulatory data drawn from speakers of American English. This corpus, comprising co-collected audio, nasal, voicing, and articulatory movement, will serve as 'ground truth' for training and assessing the capabilities of the fully trained speech inversion system. As a further test, we will test it against ground truth data from speakers of languages with patterns of articulatory habits known to differ from English.The goal of this project is to develop and refine a Speech Inversion Tool that 'reads' acoustic recordings of speech and 'recovers' details of the magnitude and timing of articulatory movements. The project aims to accomplish this goal by training specialized Neural Network models to relate features of the acoustic signal to separately acquired ground-truth nasal vs. oral outflow signals and concurrent electroglottography. Training data derives from native speakers of English; validation and tests for generalization include productions of speakers of Canadian French and Russian. When successfully validated, the resulting speech inversion tool will be useful for identifying medical issues that affect speech movement organization, such as the well-known disruption of oral/laryngeal timing in speakers with dysarthria. In addition, incorporating estimates of articulation may also aid in the tracking of changes resulting from medical conditions such as depression and schizophrenia. More generally, the ability to rapidly and easily analyze articulatory movements obtained from audio recordings alone has the potential substantially improve Automated Speech Recognition (ASR) systems, and to assist scholars, forensic scientists, and clinical professionals studying the speech of communities under field conditions in rural or under-resourced areas, and to help in the documentation of endangered languages.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.