Audiovisual integration of speech in noise: role of behavioural and neural mechanism

沟通往往发生在噪音中。当音频信号降低时，来自说话者面部和嘴巴运动的视觉线索有助于改善我们的语音感知，特别是对于那些听力损失的人。然而，神经机制支持视听整合语音噪声在很大程度上是未知的。特别是，听觉皮层附近的区域和视听整合的语音在噪声中的个人听力损失和没有之间的因果关系尚未建立。目前的项目旨在解决这一差距，并澄清行为和神经机制，支持视听语音噪声。在实验1中，被试在听视听演讲时的注视将被眼动仪监测。我们的目标是建立视觉注意，个人认知能力和言语感知之间的关系。实验2将有一个类似的设计，除了参与者将重复任务后，接受经颅磁刺激（TMS）到听觉皮层附近的区域，以揭示支持神经机制。实验3使用经颅直流电刺激（tDCS）重复实验1和2，以利用机器学习技术量化视觉注意力、个体认知能力和tDCS反应性之间的相关性。

Communication often occurs in noise. When the audio signal is degraded, visual cues from the speaker's facial and mouth movements help to improve our speech perception, especially for those with hearing loss. However, the neural mechanisms supporting audiovisual integration of speech in noise are largely unknown. In particular, the causal correlation between areas near the auditory cortex and audiovisual integration of speech in noise for individuals with and without hearing loss is not yet established. The current project aims to address this gap and clarify the behavioural and neural mechanisms supporting audiovisual speech in noise. In Experiment 1, participants' gaze will be monitored with an eye-tracker when listening to audiovisual speech. We aim to establish the relationship between visual attention, individual cognitive abilities, and speech perception. Experiment 2 will have a similar design, except participants will repeat the task after receiving Transcranial Magnetic Stimulation (TMS) to an area nearby the auditory cortex to uncover the supporting neural mechanisms. Experiment 3 repeats experiment 1 and 2 with transcranial Direct Current Stimulation (tDCS) to quantify the correlation between visual attention, individual cognitive abilities, and tDCS responsiveness with machine learning techniques.