Audiovisual Perception of Emotion and Speech in Hearing Individuals and Cochlear Implant Users

通过听觉口语进行交流的能力被视为人工耳蜗植入（CI）成功的基准，但这忽视了视觉线索在交流中的重要作用。社会情绪信号的相关性及其对CI生活质量的重要性（Luo，克恩，& Pulling，2018; Schorr，Roth，& Fox，2009）要求研究视觉对沟通的好处。招募通过面部和声音进行交流的模型（Young，Frühholz和Schweinberger，2020），我们认为耳聋可以引起跨模态皮层可塑性，例如视觉刺激可以激活听觉皮层区域。即使在适应CI后，初步研究结果表明视觉信息对言语和说话者性别的感知有特别强的贡献。需要在功能和大脑层面更好地了解这些现象，以促进有效的干预措施，改善沟通，并最终提高生活质量。在这里，我们专注于语后聋成年CI用户，并提出了四项研究（S1-S4）。在S1中，我们进行了系统的审查，以确定目前的知识状态的视觉信息（面部或手动手势）的作用，情感识别和语音感知的声音，在听力成人和CI用户。在S2中，我们探索在行为实验与动态时间同步的视听刺激是否CI用户受益于一致的面部表情时，识别声音的情绪比听力成人-这是否持有，即使在控制整体的语音只有性能水平。重要的是，我们使用语音变形技术，而不是噪音，来等同于性能水平。在S3中，我们研究了视听整合（AVI）在事件相关电位（ERP）中与视听情绪刺激的脑相关性。我们专注于一致的AV刺激，以加快神经处理的能力，并调查个体神经标记的AVI和情绪识别的行为表现之间的关系。在S4中，我们研究了用漫画化的声音情感进行知觉训练可以在多大程度上提高成年CI用户的听觉和视听情感识别。我们评估了所有研究中情绪识别能力和生活质量之间的关系。该项目建立在DFG资助的语音感知和视听整合识别说话者和语音的成功研究基础上，以及我们与图林根州的Cobriar植入康复中心的长期合作。我们希望这项工作将有助于模型的认知和大脑机制的多模态感知在人类沟通。我们建议，更好地了解视觉面部信号支持CI用户的机制将提供重要的信息，可用于优化语言和社会情感沟通，并最终提高生活质量。

The ability to communicate via auditory spoken language is taken as a benchmark for success of cochlear implants (CIs), but this disregards the important role visual cues play in communication. The relevance of socio-emotional signals and their importance for quality of life with a CI (Luo, Kern, & Pulling, 2018; Schorr, Roth, & Fox, 2009) calls for research on visual benefits to communication. Recruiting models of communication via the face and voice (Young, Frühholz, & Schweinberger, 2020), we consider that deafness can elicit crossmodal cortical plasticity, such that visual stimuli can activate auditory cortex areas. Even after adaptation to a CI, initial findings suggest a particularly strong contribution of visual information to the perception of speech and speaker gender. Better understanding of these phenomena at the functional and brain level is required to promote efficient interventions improving communication, and ultimately life quality. Here we focus on postlingually deaf adult CI users and propose four studies (S1-S4). In S1, we conduct a systematic review to determine the current state of knowledge regarding the role of visual information (face or manual gesture) for emotion recognition and speech perception from voices, in hearing adults and CI users. In S2, we explore in a behavioral experiment with dynamic time-synchronized audiovisual stimuli whether CI users benefit more from congruent facial expressions when recognizing vocal emotions than do hearing adults – and whether this holds even when controlling for overall auditory-only performance levels. Importantly, we use voice morphing technology, rather than noise, to equate performance levels. In S3, we study brain correlates of audiovisual integration (AVI) in event-related potentials (ERPs) to audiovisual (AV) emotional stimuli. We focus on the ability of congruent AV stimuli to speed up neural processing, and investigate relationships between individual neural markers of AVI and behavioral performance in emotion recognition. In S4, we study the degree to which perceptual training with caricatured vocal emotions can improve auditory and audiovisual emotion recognition in adult CI users. We assess relationships between emotion recognition abilities and reported quality of life in all studies. The project builds on successful previous research funded by the DFG on Voice Perception and Audiovisual integration in the identification of speaker and speech, and on our long-standing collaboration with the Cochlear Implant Rehabilitation Centre in Thuringia. We hope this work will contribute to models of the cognitive and brain mechanisms underlying multimodal perception in human communication. We propose that better understanding of the mechanisms by which visual facial signals support CI users will provide important information that can be used to optimize both linguistic and socio-emotional communication, and ultimately quality of life.