Audiovisual Perception of Emotion and Speech in Hearing Individuals and Cochlear Implant Users

听力正常者和人工耳蜗使用者对情绪和言语的视听感知

• 批准号: 501994399
• 负责人: Professor Dr. Christian Dobel
• 金额: --
• 依托单位: Institut für Anglistik und Amerikanistik
• 依托单位国家: 德国
• 项目类别: Priority Programmes
• 资助国家: 德国
• 项目状态: 未结题
• 来源: https://gepris.dfg.de/gepris/projekt/501994399?language=en
• 关键词: Audiovisual; Perception; Emotion; Speech; Hearing

The ability to communicate via auditory spoken language is taken as a benchmark for success of cochlear implants (CIs), but this disregards the important role visual cues play in communication. The relevance of socio-emotional signals and their importance for quality of life with a CI (Luo, Kern, & Pulling, 2018; Schorr, Roth, & Fox, 2009) calls for research on visual benefits to communication. Recruiting models of communication via the face and voice (Young, Frühholz, & Schweinberger, 2020), we consider that deafness can elicit crossmodal cortical plasticity, such that visual stimuli can activate auditory cortex areas. Even after adaptation to a CI, initial findings suggest a particularly strong contribution of visual information to the perception of speech and speaker gender. Better understanding of these phenomena at the functional and brain level is required to promote efficient interventions improving communication, and ultimately life quality. Here we focus on postlingually deaf adult CI users and propose four studies (S1-S4). In S1, we conduct a systematic review to determine the current state of knowledge regarding the role of visual information (face or manual gesture) for emotion recognition and speech perception from voices, in hearing adults and CI users. In S2, we explore in a behavioral experiment with dynamic time-synchronized audiovisual stimuli whether CI users benefit more from congruent facial expressions when recognizing vocal emotions than do hearing adults – and whether this holds even when controlling for overall auditory-only performance levels. Importantly, we use voice morphing technology, rather than noise, to equate performance levels. In S3, we study brain correlates of audiovisual integration (AVI) in event-related potentials (ERPs) to audiovisual (AV) emotional stimuli. We focus on the ability of congruent AV stimuli to speed up neural processing, and investigate relationships between individual neural markers of AVI and behavioral performance in emotion recognition. In S4, we study the degree to which perceptual training with caricatured vocal emotions can improve auditory and audiovisual emotion recognition in adult CI users. We assess relationships between emotion recognition abilities and reported quality of life in all studies. The project builds on successful previous research funded by the DFG on Voice Perception and Audiovisual integration in the identification of speaker and speech, and on our long-standing collaboration with the Cochlear Implant Rehabilitation Centre in Thuringia. We hope this work will contribute to models of the cognitive and brain mechanisms underlying multimodal perception in human communication. We propose that better understanding of the mechanisms by which visual facial signals support CI users will provide important information that can be used to optimize both linguistic and socio-emotional communication, and ultimately quality of life.

通过听觉语言进行交流的能力被视为人工耳蜗植入（CIs）成功的基准，但这忽视了视觉线索在交流中的重要作用。社会情感信号的相关性及其对CI生活质量的重要性（Luo, Kern, & pull2018; Schorr, Roth, & Fox, 2009）要求研究视觉对沟通的好处。招募通过面部和声音进行交流的模型（Young, fr<s:2> hholz, & Schweinberger, 2020），我们认为耳聋可以引发皮层的跨模态可塑性，因此视觉刺激可以激活听觉皮层区域。即使在适应了CI之后，最初的研究结果表明，视觉信息对语音和说话者性别的感知有特别强的贡献。我们需要在功能和大脑层面更好地理解这些现象，以促进有效的干预措施，改善沟通，最终提高生活质量。在这里，我们将重点放在语言后失聪的成人CI用户上，并提出了四项研究（S1-S4）。在S1中，我们进行了一项系统综述，以确定视觉信息（面部或手势）在听力正常的成年人和CI用户中对声音的情感识别和语音感知中的作用的知识现状。在S2中，我们通过动态时间同步视听刺激的行为实验，探讨了CI用户在识别声音情绪时是否比听力正常的成年人更能从一致的面部表情中获益，以及即使在控制整体听觉表现水平时，这一结果是否也适用。重要的是，我们使用语音变形技术，而不是噪音，来平衡性能水平。在S3中，我们研究了事件相关电位（ERPs）对视听（AV）情绪刺激的视听整合（AVI）的脑相关性。本研究聚焦于一致的视音频刺激加速神经加工的能力，并探讨了视音频个体神经标记与情绪识别行为表现之间的关系。在S4中，我们研究了具有漫画声音情绪的感知训练在多大程度上可以提高成人CI用户的听觉和视听情感识别。我们评估了所有研究中情绪识别能力和报告生活质量之间的关系。该项目建立在之前由DFG资助的关于声音感知和视听整合识别说话者和言语的成功研究基础上，以及我们与图林根州人工耳蜗康复中心的长期合作基础上。我们希望这项工作将有助于建立人类交流中多模态感知的认知和大脑机制模型。我们建议，更好地理解视觉面部信号支持CI用户的机制，将提供重要的信息，可用于优化语言和社会情感沟通，并最终提高生活质量。