UNS: Collaborative Research: Prosodic Control of Speech Synthesis for Assistive Communication in Severe Paralysis

UNS：合作研究：语音合成的韵律控制用于严重瘫痪的辅助沟通

• 批准号: 1509791
• 负责人: Susan Koch Fager
• 金额: $ 11.2万
• 依托单位: Madonna Rehabilitation Hospital
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-07-15 至 2018-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1509791&HistoricalAwards=false
• 关键词: UNS; Collaborative; Research; Prosodic; Control

1510563(Stepp) & 1509791 (Koch Fager)This work will develop and evaluate a system to allow individuals with unintelligible speech due to severe paralysis to control a speech synthesizer that includes prosody (changes in the pitch, loudness, and duration in speech that convey meaning). This advancement to the synthetic speech and the ease of its control by users will facilitate improved functionality of clinical communication systems, thus improving the quality of life of users. Natural and intelligible speech production in these individuals will increase their ability to participate actively in society and empower them to self-advocate for their own medical management.The research objective of this proposal is to test the hypothesis that providing users of alternative and augmentative communication (AAC) with a method for prosodic control will result in speech synthesis that is more natural to listeners and provides greater function to users. Up to 1.2% of the population is unable to meet daily communication needs using typical speech due to stroke or other neurological injury, requiring AAC to meet their communication needs. Their quality of life is strongly dependent on access to this communication, both for social interaction as well as to relay information about urgent medical needs. The most advanced AAC devices incorporate speech synthesis, allowing the users to communicate orally with others. However, the resulting synthetic speech is both unnatural and difficult for others to understand, and is often described as "robotic". Specifically, synthetic speech does not vary in pitch, loudness, or rhythm, the prosodic features utilized in typical speech to relay emotional state, utterance form (statement vs. question), irony, and emphasis.Asking AAC users to control each of these dimensions individually would result in an intractably slow and complex system, an unacceptable burden for individuals who already have considerably reduced communication rates. Instead, this project will leverage the fact that typical speech predictably uses these prosodic markers (pitch, loudness, rhythm) in concert. A novel AAC interface will be developed to allow users to modify the overall "stress" of synthetic speech output as a single dimension, in order to provide easily controlled, natural, and intelligible speech synthesis. The co-PIs will use their combined expertise in speech technology, clinical application of AAC, and real-time control of human-machine-interfaces to enable essential advancements in AAC technology to achieve three goals. In Research Goal 1, a multi-stress speech bank for concatenative speech synthesis will be created via a novel interactive procedure in which speech productions of healthy speakers are "misunderstood", thus prompting speakers to naturally emphasize specific target sounds in their repeated responses. This will result in a bank of triphones (sounds with a specific left and right context, based on surrounding sounds) with all potential combinations of sounds and stresses. Research Goal 2 is to develop an AAC interface that allows users to select phonemes (individual sounds of speech) using two-dimensional cursor control (e.g., head-tracking, eye-tracking) in which the stress of individual phonemes will be based on cursor dwell time. In Research Goal 3, the functionality of the AAC interface will be evaluated by testing its effect on the naturalness of communicative interactions.

1510563(STEPP)&AMP；1509791(科赫·法格)这项工作将开发和评估一个系统，允许由于严重瘫痪而无法理解语音的个人控制包括韵律(传达意思的语音中的音高、响度和持续时间的变化)的语音合成器。这种对合成语音的改进和用户对其控制的易用性将促进临床通信系统的功能改进，从而提高用户的生活质量。这些人自然、易懂的语音产生将提高他们积极参与社会的能力，并使他们能够自我倡导自己的医疗管理。本提案的研究目的是检验这样一种假设，即为替代和增强通信(AAC)的用户提供一种韵律控制方法将导致对听者更自然的语音合成，并为用户提供更大的功能。由于中风或其他神经损伤，高达1.2%的人口无法使用典型的语言满足日常交流需求，这需要AAC来满足他们的交流需求。他们的生活质量在很大程度上依赖于这种通信，无论是在社交方面，还是在传递有关紧急医疗需求的信息方面。最先进的AAC设备包含语音合成，允许用户与他人进行口头交流。然而，由此产生的合成语音既不自然，也很难被其他人理解，经常被描述为“机器人”。具体地说，合成语音在音调、响度或节奏上没有变化，这些韵律特征在典型语音中用来传达情绪状态、话语形式(陈述与问题)、讽刺和强调。让AAC用户单独控制这些维度将导致一个难以处理的缓慢和复杂的系统，对于已经大幅降低通信速率的个人来说，这是一个不可接受的负担。相反，这个项目将利用这样一个事实，即典型的语音可以预测地使用这些韵律标记(音调、响度、节奏)。将开发一种新颖的AAC接口，以允许用户将合成语音输出的总体“重音”修改为单一维度，以便提供容易控制的、自然的和可理解的语音合成。联合PIS将利用他们在语音技术、AAC临床应用和人机界面实时控制方面的综合专业知识，使AAC技术取得重大进步，以实现三个目标。在研究目标1中，将通过一种新颖的互动程序创建一个用于串联语音合成的多重音语音库，在该过程中，健康说话者的语音产物将被“误解”，从而促使说话者在其重复反应中自然地强调特定的目标声音。这将产生一组三音节(具有特定左右上下文的声音，基于周围的声音)，其中包含所有可能的声音和重音组合。研究目标2是开发一种AAC接口，其允许用户使用二维光标控制(例如，头部跟踪、眼睛跟踪)来选择音素(单独的语音)，其中各个音素的重音将基于光标停留时间。在研究目标3中，将通过测试AAC接口对交流互动的自然性的影响来评估AAC接口的功能。