CHS: Small: Compounding Dividends on Voice Banking

文本到语音（TTS）合成已经成为一项成功的、普遍存在的技术。激发本研究的TTS技术的应用领域是它在增强和替代通信（AAC）中的应用。根据美国言语语言和听力协会（ASHA）的数据，美国有超过200万人患有严重的沟通障碍，损害了他们说话的能力。使用TTS创建语音输出的AAC设备被许多这样的人用来支持通信。从历史上看，AAC用户只能使用相对较少的通用TTS语音，这些语音既不是他们独有的，也不适合他们的年龄或方言。然而，TTS技术的进步使得创造个性化的合成声音成为可能，如果AAC设备用户能够记录足够的语音，就可以捕捉到他们独特的声音身份。这使得患有ALS等神经退行性疾病的患者能够“储存”他们的声音——也就是说，在疾病发展到不能再说话的地步之前，记录下他们的语音样本，以后可以用来创建个人的TTS声音。不幸的是，语音银行的一个主要障碍，特别是对于那些可能已经经历了一些说话困难的患者来说，是创造一个自然的TTS声音所需的语音量，以充分捕捉语音银行的声音身份。为了减少这一障碍，本研究将结合几十年前开发的一种称为并行形成峰合成的语音合成技术，与深度学习计算技术相结合，该技术允许计算机学习如何控制并行形成峰合成器的参数，以在给定目标说话者的语音示例的情况下再现目标说话者的语音。将实现并训练一个并行的形成峰合成器来模拟语音银行家记录的语音，并将其输出与使用相同语音数据训练的其他合成器的输出进行比较。将使用合成话语和自然话语之间的客观相似性度量，以及使用人类听众的语音质量和相似性的主观度量。这将是构建基于并行形成峰合成的语音转换系统的第一步，该系统能够从少量自然语音样本中创建TTS语音，并且能够更好地模拟自然语音的表达性质。尽管TTS技术取得了进步，但该技术在语音银行中的应用仍面临诸多挑战。具体来说：(a)使用单元选择或混合DNN/单元选择创建最自然的TTS语音所需的语音量（几个小时）对大多数语音银行家来说是令人望而却步的；(b)与串联合成相比，现有的语音转换技术不需要大量的目标说话者的平行语音，通常会产生听起来不太自然和不太像目标说话者的语音；(c)连接和统计参数技术产生的语音仅与构建或训练它们的语音语料库中的数据一样具有表现力。平行形成峰合成，因为它明确地基于自然语音的感知上最显著的特征，并使其适合独立建模喉部，超节段和节段特征，应该能够更好地解决所有这三个挑战。作为概念验证，将实现一个基于dnn参数估计的并行形成峰合成（PFS）声码器。声码器将在Merlin DNN合成框架内实现，这样PFS系统的语音输出可以直接与World和MagPhase声码器产生的输出进行比较。培训将基于从同一组1600个语音中提取的语料库，这些语料库由多个个人录制，这些人将他们的录音贡献给ModelTalker项目。选择的目标说话者将根据性别和广泛的英语方言进行平衡，但将避免使用具有明显构音障碍水平的说话者。客观比较将基于梅尔-倒谱差异（MCD）的合成和自然句子的标记，没有使用在训练合成器。主观测量（平均意见分数）将通过亚马逊土耳其机器人从人类听众那里获得。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Text to speech (TTS) synthesis has become a successful and ubiquitous technology. The area of application for TTS technology that motivates this research is its use for Augmentative and Alternative Communication (AAC). According to the American Speech-Language and Hearing Association (ASHA), more than two million people in the United States have severe communication disorders that impair their ability to talk. AAC devices that use TTS to create spoken output are used by many of these people to support communication. Historically, AAC users have had access to a relatively small family of generic TTS voices that are neither unique to them nor typically age- or dialect-appropriate. However, advances in TTS technology make it possible to create personalized synthetic voices that capture the unique vocal identity of AAC device users if they are able to record enough speech. This allows patients with neurodegenerative diseases such as ALS to "bank" their voice - that is, to record examples of their speech that can later be used to create a personal TTS voice - before the disease progresses to a point that they can no longer speak. Unfortunately, one major barrier to voice banking, especially for patients who may already be experiencing some difficulty speaking, is the amount of speech needed to create a natural sounding TTS voice that fully captures the vocal identity of the voice banker. To reduce this barrier, this research will combine a type of speech synthesis called parallel formant synthesis that was developed several decades ago, with deep learning computational techniques that allow a computer to learn how to control the parameters of the parallel formant synthesizer to reproduce the speech of a target speaker given examples of the target speaker's speech. A parallel formant synthesizer will be implemented and trained to model speech recorded by voice bankers, and its output will be compared with that of other synthesizers that have been trained with the same speech data. Objective measures of similarity between synthetic and natural utterances, and subjective measures of voice quality and similarity using human listeners, will be used. This will be the first step toward building a parallel formant synthesis-based voice conversion system capable of creating TTS voices from a small number of natural speech samples, and also better able to model the expressive nature of natural speech.Despite advances in TTS technology, there are multiple challenges to the application of this technology for voice banking. Specifically: (a) the amount of speech required (several hours) to create the most natural sounding TTS voices using unit selection or hybrid DNN/unit selection is prohibitive for most voice bankers; (b) existing voice conversion techniques that do not require large amounts of parallel speech from the target talker generally produce speech sounding less natural and less like the target speaker when compared to concatenative synthesis; and (c) both concatenative and statistical parametric techniques produce speech that is only as expressive as the data within the speech corpus from which they have been constructed or trained. Parallel formant synthesis, because it is based explicitly on the perceptually most salient features of natural speech and lends itself to independently modeling laryngeal, suprasegmental, and segmental features should be better able to address all three of these challenges. As proof of concept, a parallel formant synthesis (PFS) vocoder with DNN-based parameter estimation will be implemented. The vocoder will be implemented within the Merlin DNN synthesis framework so that speech output of the PFS system can be directly compared to output generated by the World and MagPhase vocoders. Training will be based on corpora drawn from the same set of 1600 utterances recorded by multiple individuals who have contributed their recordings to the ModelTalker project. The selected target talkers will be balanced for gender and span a wide range of English dialects, but use of speakers with noticeable levels of dysarthria will be avoided. Objective comparisons will be based on Mel-Cepstral Difference (MCD) between synthetic and natural sentence tokens that were not used in training the synthesizers. Subjective measures (Mean Opinion Scores) will be obtained from human listeners via Amazon Mechanical Turk.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.