UNS: Collaborative Research: Prosodic Control of Speech Synthesis for Assistive Communication in Severe Paralysis

1510563(Stepp) & &; 1509791 （Koch Fager）这项工作将开发和评估一个系统，使由于严重瘫痪而无法理解语言的人能够控制包括韵律（通过音调、音量和持续时间的变化来传达意思）的语音合成器。这种合成语音的进步及其易于用户控制将有助于改善临床通信系统的功能，从而提高用户的生活质量。这些人的自然和可理解的语言生产将提高他们积极参与社会的能力，并使他们能够自我倡导自己的医疗管理。本提案的研究目的是验证假设，即为替代和增强通信（AAC）的用户提供一种韵律控制方法，将导致语音合成对听者来说更自然，并为用户提供更大的功能。由于中风或其他神经损伤，高达1.2%的人口无法使用典型语言满足日常沟通需求，需要AAC来满足他们的沟通需求。他们的生活质量在很大程度上依赖于这种交流，既为了社会交往，也为了传递有关紧急医疗需求的信息。最先进的AAC设备包含语音合成，允许用户与他人口头交流。然而，由此产生的合成语音既不自然，也很难让其他人理解，通常被描述为“机器人”。具体来说，合成语音在音高、响度或节奏上没有变化，在典型语音中利用的韵律特征来传递情绪状态、话语形式（陈述句与疑问句）、讽刺和强调。要求AAC用户单独控制这些维度将导致系统变得非常缓慢和复杂，对于已经大大降低了通信速率的用户来说，这是一个不可接受的负担。相反，这个项目将利用这样一个事实，即典型的语音可以预测地使用这些韵律标记（音高、响度、节奏）。一种新的AAC接口将被开发，允许用户将合成语音输出的整体“应力”修改为单一维度，以提供易于控制，自然和可理解的语音合成。合作pi将利用他们在语音技术，AAC临床应用和人机界面实时控制方面的综合专业知识，使AAC技术的基本进步实现三个目标。在研究目标1中，我们将通过一种新颖的交互过程来创建一个用于串联语音合成的多重音语音库，在这个过程中，健康说话者的语音产生被“误解”，从而促使说话者在重复的回答中自然地强调特定的目标音。这样就会产生一组三音音（基于周围声音的特定左右语境的声音），其中包含所有可能的声音和重音组合。研究目标2是开发一个AAC接口，允许用户使用二维光标控制（例如头部跟踪，眼球跟踪）来选择音素（单个语音），其中单个音素的压力将基于光标停留时间。在研究目标3中，AAC接口的功能将通过测试其对交流交互的自然性的影响来评估。

1510563(Stepp) & 1509791 (Koch Fager)This work will develop and evaluate a system to allow individuals with unintelligible speech due to severe paralysis to control a speech synthesizer that includes prosody (changes in the pitch, loudness, and duration in speech that convey meaning). This advancement to the synthetic speech and the ease of its control by users will facilitate improved functionality of clinical communication systems, thus improving the quality of life of users. Natural and intelligible speech production in these individuals will increase their ability to participate actively in society and empower them to self-advocate for their own medical management.The research objective of this proposal is to test the hypothesis that providing users of alternative and augmentative communication (AAC) with a method for prosodic control will result in speech synthesis that is more natural to listeners and provides greater function to users. Up to 1.2% of the population is unable to meet daily communication needs using typical speech due to stroke or other neurological injury, requiring AAC to meet their communication needs. Their quality of life is strongly dependent on access to this communication, both for social interaction as well as to relay information about urgent medical needs. The most advanced AAC devices incorporate speech synthesis, allowing the users to communicate orally with others. However, the resulting synthetic speech is both unnatural and difficult for others to understand, and is often described as "robotic". Specifically, synthetic speech does not vary in pitch, loudness, or rhythm, the prosodic features utilized in typical speech to relay emotional state, utterance form (statement vs. question), irony, and emphasis.Asking AAC users to control each of these dimensions individually would result in an intractably slow and complex system, an unacceptable burden for individuals who already have considerably reduced communication rates. Instead, this project will leverage the fact that typical speech predictably uses these prosodic markers (pitch, loudness, rhythm) in concert. A novel AAC interface will be developed to allow users to modify the overall "stress" of synthetic speech output as a single dimension, in order to provide easily controlled, natural, and intelligible speech synthesis. The co-PIs will use their combined expertise in speech technology, clinical application of AAC, and real-time control of human-machine-interfaces to enable essential advancements in AAC technology to achieve three goals. In Research Goal 1, a multi-stress speech bank for concatenative speech synthesis will be created via a novel interactive procedure in which speech productions of healthy speakers are "misunderstood", thus prompting speakers to naturally emphasize specific target sounds in their repeated responses. This will result in a bank of triphones (sounds with a specific left and right context, based on surrounding sounds) with all potential combinations of sounds and stresses. Research Goal 2 is to develop an AAC interface that allows users to select phonemes (individual sounds of speech) using two-dimensional cursor control (e.g., head-tracking, eye-tracking) in which the stress of individual phonemes will be based on cursor dwell time. In Research Goal 3, the functionality of the AAC interface will be evaluated by testing its effect on the naturalness of communicative interactions.