The effect of gaze on EEG measures of multisensory integration in a cocktail party scenario.
The effect of gaze on EEG measures of multisensory integration in a cocktail party scenario.
复制标题
鸡尾酒会场景中凝视对多感官整合脑电图测量的影响。
DOI：
10.1101/2023.08.23.554451
复制
发表时间：
2023
期刊：
影响因子：
--
通讯作者：
Lalor,EdmundC
中科院分区：
文献类型：
--
作者：
Ahmed,Farhin;Nidiffer,AaronR;Lalor,EdmundC
在嘈杂的环境中，看到说话人的脸可以大大提高我们的言语理解能力。这是因为大脑有能力将我们周围的听觉和视觉信息联合收割机，这一过程被称为多感觉整合。选择性注意力也会强烈影响我们在多人说话的场景中的理解这种效应被称为鸡尾酒会现象。然而，注意力和多感官整合之间的相互作用还没有完全理解，特别是在自然，连续的语音方面。在最近的脑电图（EEG）的研究，我们探讨了这个问题，并表明，多感官整合时，视听扬声器出席相比，当扬声器是无人值守。在这里，我们将这项工作扩展到研究这种互动如何根据一个人的凝视行为而变化，这会影响他们所获得的视觉信息的质量。要做到这一点，我们记录了EEG从31名健康的成年人，因为他们进行选择性注意力的任务，在几个范例，涉及两个同时提出的视听扬声器。然后，我们对所记录的EEG如何与所呈现的扬声器的音频语音（包络）相关进行建模。至关重要的是，我们比较了两类模型-一个假设潜在的多感觉整合（AV）与另一个假设两个独立的unisensory听觉和视觉过程（A+V）。这种比较揭示了当参与者直视视听演讲者的脸时，对多感觉整合的强烈注意力影响的证据。当演讲者的脸在参与者的周边视觉中时，这种效果并不明显。总的来说，我们的研究结果表明，当高保真度的视觉（发音）语音信息时，注意力对多感觉整合有很大的影响。更一般地说，这表明，在自然视听语音过程中，注意力和多感官整合之间的相互作用是动态的，并且可以根据特定的任务和环境进行调整。
Seeing the speaker’s face greatly improves our speech comprehension in noisy environments. This is due to the brain’s ability to combine the auditory and the visual information around us, a process known as multisensory integration. Selective attention also strongly influences what we comprehend in scenarios with multiple speakers–an effect known as the cocktail-party phenomenon. However, the interaction between attention and multisensory integration is not fully understood, especially when it comes to natural, continuous speech. In a recent electroencephalography (EEG) study, we explored this issue and showed that multisensory integration is enhanced when an audiovisual speaker is attended compared to when that speaker is unattended. Here, we extend that work to investigate how this interaction varies depending on a person’s gaze behavior, which affects the quality of the visual information they have access to. To do so, we recorded EEG from 31 healthy adults as they performed selective attention tasks in several paradigms involving two concurrently presented audiovisual speakers. We then modeled how the recorded EEG related to the audio speech (envelope) of the presented speakers. Crucially, we compared two classes of model – one that assumed underlying multisensory integration (AV) versus another that assumed two independent unisensory audio and visual processes (A+V). This comparison revealed evidence of strong attentional effects on multisensory integration when participants were looking directly at the face of an audiovisual speaker. This effect was not apparent when the speaker’s face was in the peripheral vision of the participants. Overall, our findings suggest a strong influence of attention on multisensory integration when high fidelity visual (articulatory) speech information is available. More generally, this suggests that the interplay between attention and multisensory integration during natural audiovisual speech is dynamic and is adaptable based on the specific task and environment.