Cortical and behavioural measures of active communication

我们对听觉语音处理和注意力的理解主要源于受控的实验室实验，这些实验使用了简化的人工刺激，其复杂性和可预测性与现实场景不同。虽然唇读等视觉线索对听觉语音处理的影响是众所周知的，但在社会互动等复杂交流场景中的视听处理却没有受到太多关注。现实生活中的交流包括身体动作、手势、面部表情和眨眼等视觉线索的互动循环，这些在传统的实验室实验中无法很好地捕捉到，这些实验将倾听者视为被动的接受者。我们开发交互式虚拟环境（VEs）来推进该领域，因为VEs提供了实现和使用可复制的真实视听场景的机会。我们的ve将可复制性和再现性的目标与日常通信需求的复杂性结合起来。我们的目标是通过开发和评估模拟至少一些自然交流情况的复杂性并将听者作为主动代理的虚拟语音来提高我们对视听语音处理的理解。在神经层面上，听觉对言语注意的时间动态可以通过脑电图（EEG）非侵入性地探索。在这个项目的第一阶段，我们在非交互式的虚拟现实中实施了脑电皮层语音跟踪，使我们能够客观地评估专注的听力。具体来说，我们发现大脑皮层的语音追踪对于在虚拟现实中呈现的没有脚本但预先录制好的故事产生了有效的结果。在这个项目的第二阶段，我们将推进我们的虚拟现实模拟真实和远程呈现的条件。这些虚拟环境支持对实时、交互式通信场景的研究。在研究1中，我们将在二元交流场景中比较被动倾听和主动对话，并操纵对话参与（主动与被动）和对话者的视觉表现（真实在场与远程在场）。在研究2中，我们将使用三向轮流交流场景来探讨熟悉和不熟悉对话者之间的听觉加工和人际同步。我们将引入对话者熟悉度的因素，并研究视觉对话者表征（真实在场与远程在场）对听觉加工和人际同步的影响。研究3将探讨在视听干扰情境下，听觉加工和人际同步，从而操纵沟通困难。在这三项研究中，我们将使用脑电图作为一种非侵入性手段来监测对语音和非语音信号的注意。此外，行为同步的措施之间的谈话个人将开发预测沟通努力。

Our understanding of auditory speech processing and attention mainly stems from controlled laboratory experiments using simplified, artificial stimuli that differ in complexity and predictability from real-life scenarios. While the influence of visual cues such as lip-reading on auditory speech processing is well known, audio-visual processing in complex communication scenarios such as social interactions has not received much attention. Real-life communication involves interactive loops with visual cues like body movements, gestures, facial expressions, and eye blinks, which are not well captured in traditional lab-based experiments that treat a listening person as a passive receiver. We develop interactive virtual environments (VEs) to advance the field, as VEs offer the opportunity to implement and use reproducible realistic audio-visual scenarios. Our VEs combine the goals of replicability and reproducibility with the complexity of everyday communication demands. Our aim is to advance our understanding of audio-visual speech processing by developing and evaluating VEs that mimic at least some of the complexity of natural communication situations and include the listener as an active agent. On a neural level, the temporal dynamics of auditory attention to speech can be non-invasively explored by means of electroencephalography (EEG). Cortical speech tracking with EEG was implemented in the first period of this project in non-interactive VEs and allowed us to objectively evaluate attentive listening. Specifically, we found that cortical speech tracking produced valid results for unscripted but pre-recorded stories presented in a VE. In the second period of this project, we will advance our VEs to simulate real and telepresence conditions. These VEs enable the study of live, interactive communication scenarios. In Study 1, we will compare passive listening with active conversation in a dyadic communication scenario and manipulate conversation involvement (active vs. passive) and visual representation of interlocutors (real presence vs. telepresence). In Study 2, we will explore auditory processing and interpersonal synchrony between familiar and unfamiliar interlocutors using triadic turn-taking communication scenarios. We will introduce the factor of interlocutor familiarity and investigate the influence of visual interlocutor representation (real presence vs. telepresence) on auditory processing and interpersonal synchrony. Study 3 will investigate auditory processing and interpersonal synchrony in triadic turn-taking situations in the presence of audio-visual distractor scenarios, thereby manipulating communication difficulty. In all three studies, we will use EEG as a non-invasive means of monitoring attention to speech and non-speech signals. In addition, measures of behavioral synchrony between conversing individuals will be developed to predict communication effort.