Audiovisual speech perception in cochlear implant recipients

语音感知本质上是多模态的。除了听觉线索，视觉特征也提供了重要的信息。这在人工耳蜗使用者中尤其明显，他们可能表现出特别好的语言阅读能力。然而，在临床常规和研究中，CI受者的言语感知评估通常仅是听觉方面的，这可能至少部分是由于缺乏合适的视听测试材料。然而，考虑到听觉和视觉语言将允许更高的生态有效性的测量。本项目提出了一种独特的方法，即通过基于计算机的发音模型（“说话头”）呈现视觉线索来评估视听语音。这种方法有几个重要的优点。首先，传统的语音听力测量材料仍然可以使用，因为它们被说话头提供的视觉线索所补充。其次，发音模型提供了呈现任意视觉线索（一致、不一致、不同显著性）的机会，从而允许使用高度控制的视听刺激。该计划旨在建立各种视听语言材料，适合于CI接受者的语音感知的临床和科学评估。基于这些材料，研究了不同的主题，如植入后多模态语音感知的发展。此外，跨模态重组是通过呈现一致和不一致的视听信息和比较听觉皮层在不同语音处理水平上的相互作用来解决的。多说话者语音识别将通过考虑说话者特征和视觉语音线索的影响来详细研究，这些线索允许分离竞争语音流。韵律感知将通过解决视觉语音线索在句子重音感知中的作用来研究。此外，还考虑了诸如双边与双峰CI使用之类的拟合方面。由于CI接受者的多模态语音感知尚未得到全面检查，并且通过发音模型呈现高度控制和标准化视觉线索的独特方法，预计该项目将具有重大的临床和科学影响，可以详细了解生态有效条件下CI介导的语音感知。

Speech perception is multimodal in nature. Besides auditory cues visual features give important information. This is especially evident in cochlear implant (CI) users who might reveal particularly good speech reading ability. Nevertheless, assessment of speech perception in CI recipients typically is solely auditory, both in clinical routine and in research, which might at least in part be due to a lack in suitable audiovisual test materials. However, considering both auditory and visual speech would allow higher ecological validity of the measurements.This project proposes a unique approach, namely the assessment of audiovisual speech by means of presenting visual cues via a computer-based articulation model ("talking head"). This approach has several important advantages. First, conventional speech audiometric materials can still be used since they are supplemented by the visual cues provided by the talking head. Second, the articulation model gives the opportunity of presenting arbitrary visual cues (congruent, incongruent, different salience) and thus allows using highly controlled audiovisual stimuli. The proposed project aims at establishing various audiovisual speech materials suited for both clinical and scientific assessment of speech perception in CI recipients. Based on these materials different topics are considered, such as the development of multimodal speech perception after implantation. Moreover, cross-modal reorganization is addressed by presenting congruent and incongruent audiovisual information and comparing interactions in the auditory cortex across different levels of speech processing. Multi-talker speech recognition will be examined in detail by considering effects of talker characteristics and visual speech cues that allow segregation of competing speech streams. Prosody perception will be investigated by addressing the role of visual speech cues in sentence stress perception. Moreover, fitting aspects such as bilateral versus bimodal CI use are taken into account.Due to the fact that multimodal speech perception in CI recipients has not yet been comprehensively examined and the unique approach of presenting highly controlled and standardized visual cues by an articulation model it is expected that the project will have significant clinical and scientific impact allowing detailed insight into CI-mediated speech perception under ecologically valid conditions.