Generating Personalized Synthetic Speech for Progressive Dysarthria Using Severity-Appropriate Adaptation Strategies for Neural Text-to-Speech and Voice Conversion

项目摘要 超过200万美国人患有复杂的沟通障碍，损害了他们的说话能力。损失 语言障碍是肌萎缩侧索硬化症（ALS）等神经系统疾病最具破坏性的影响之一， 95%的人会逐渐失去说话的能力，陷入孤立的状态。通信 具有电子语音输出的设备允许患者增强或代替口头交流作为他们的讲话 恶化。这些设备上可用的文本（字母表、消息）可直接使用功能访问 身体部位（手指、头部、眼睛），所选文本通过文本到语音（TTS）转换为语音 技术.在当前设备上可用的电子TTS语音在年龄、性别和/或功能方面具有有限的选项。 方言，这减少了一个真正的话语的经验，因为无论是用户还是他们的 通信伙伴可以涉及设备语音。声音是一个人身份的组成部分， 一个捕捉到这种身份的声音，用户往往会退出互动，大大降低他们的生活质量， 并导致该技术的低接受度。个性化TTS语音选项是ALS的关键需求 这是一个很好的机会，让他们能够在面对重大生活变化时自由交流。 本研究的长期目标是基于软件的高性能个性化语音合成， 由有沟通障碍的人在移动的平台和商业语音设备上使用。我们 短期目标是研究利用最先进的端到端神经TTS的创新方法， 为那些已经表现出语言损失的人生成可理解的，自然的和个性化的合成语音。 人症神经TTS技术的表现明显优于前几代TTS技术， 开发高质量TTS系统的障碍。虽然使用神经TTS显然是可取的，但对大容量TTS的需求仍然存在。 大量的高质量语音数据禁止直接为ALS患者训练这样的系统。我们解决 这个问题通过我们在这个探索性项目中的两个具体目标来解决：（i）通过使用语音来适应神经TTS输出 转换为ALS的个性化TTS语音选项，以及（ii）调整神经TTS输入特征和网络 参数来个性化ALS的TTS语音选项。我们的方法为这两个目标将保留TTS语音 可理解性和自然性，同时增强语音相似性，通过使用适量的语音数据， ALS患者。 我们的适应神经TTS系统预计将产生个性化的合成语音， 个体ALS用户的特征沿着以及可理解性和自然性，以促进沟通， 倾听的安慰该项目的目标与NIH-NIDCD的优先领域“推进小说研究”相一致 增强和替代性沟通方法”。项目成果预计将 提供了大量患有不同病因的交流障碍（ALS，中风， 创伤）与个性化的声音表达和社会身份。

PROJECT SUMMARY More than 2 million Americans have a complex communication disorder that impairs their ability to talk. The loss of speech is among the most debilitating effects of neurological diseases like amyotrophic lateral sclerosis (ALS), where 95% will progressively lose their ability to speak and get trapped in a state of isolation. Communication devices with electronic voice output allow patients to augment or replace verbal communication as their speech deteriorates. The text (alphabet, messages) available on these devices is accessed directly using functioning body parts (fingers, head, eyes), and the selected text is converted to speech through text-to-speech (TTS) technology. Electronic TTS voices available on current devices have limited options in terms of age, sex, and/or dialect, which diminishes the experience of a genuine discourse because neither the user nor their communication partner can relate to the device voice. Voice is an integral part of a person’s identity and without a voice that captures this identity, users tend to withdraw from interactions, greatly reducing their quality of life, and leading to low acceptance of the technology. Personalized TTS voice options are a critical need for the ALS population in order for them to be able to communicate freely in the face of major life changes. The long-term goal of this research is software-based, high-performance personalized speech synthesis that can be used on mobile platforms and commercial speech devices by people with communication disorders. Our short-term goal is to investigate innovative methods that leverage state-of-the-art, end-to-end neural TTS, to generate intelligible, natural, and personalized synthetic speech for people who already exhibit speech loss from ALS. Neural TTS has significantly outperformed the previous generations of TTS technology, and has lowered the barrier to develop high-quality TTS systems. While it is clearly desirable to use neural TTS, the need for large quantities of high-quality speech data prohibits training such a system directly for those with ALS. We address this problem through our two specific aims in this exploratory project: (i) adapt neural TTS output by using voice conversion to personalize TTS voice options for ALS and (ii) adapt neural TTS input features and network parameters to personalize TTS voice options for ALS. Our methods for both aims will preserve TTS speech intelligibility and naturalness while enhancing voice similarity, by using modest amounts of speech data from persons with ALS. Our adapted neural TTS system is expected to generate personalized synthetic speech that has the voice characteristics of individual ALS users along with intelligibility and naturalness to promote communication and listening comfort. The project goals align with NIH-NIDCD’s priority area related to “Advancing Research in Novel Augmentative and Alternative Communication (AAC) Approaches”. The project outcomes are expected to provide a significant number of people who have communication disorders from varying etiologies (ALS, stroke, trauma) with personalized vocal expression and social identity.