Cortical and behavioural measures of active communication

主动沟通的皮层和行为测量

• 批准号: 444761144
• 负责人: Professor Dr. Stefan Debener
• 金额: --
• 依托单位: Abteilung Medizinische Physik
• 依托单位国家: 德国
• 项目类别: Priority Programmes
• 资助国家: 德国
• 项目状态: 未结题
• 来源: https://gepris.dfg.de/gepris/projekt/444761144?language=en
• 关键词: Cortical; behavioural; measures; active; communication

Our understanding of auditory speech processing and attention mainly stems from controlled laboratory experiments using simplified, artificial stimuli that differ in complexity and predictability from real-life scenarios. While the influence of visual cues such as lip-reading on auditory speech processing is well known, audio-visual processing in complex communication scenarios such as social interactions has not received much attention. Real-life communication involves interactive loops with visual cues like body movements, gestures, facial expressions, and eye blinks, which are not well captured in traditional lab-based experiments that treat a listening person as a passive receiver. We develop interactive virtual environments (VEs) to advance the field, as VEs offer the opportunity to implement and use reproducible realistic audio-visual scenarios. Our VEs combine the goals of replicability and reproducibility with the complexity of everyday communication demands. Our aim is to advance our understanding of audio-visual speech processing by developing and evaluating VEs that mimic at least some of the complexity of natural communication situations and include the listener as an active agent. On a neural level, the temporal dynamics of auditory attention to speech can be non-invasively explored by means of electroencephalography (EEG). Cortical speech tracking with EEG was implemented in the first period of this project in non-interactive VEs and allowed us to objectively evaluate attentive listening. Specifically, we found that cortical speech tracking produced valid results for unscripted but pre-recorded stories presented in a VE. In the second period of this project, we will advance our VEs to simulate real and telepresence conditions. These VEs enable the study of live, interactive communication scenarios. In Study 1, we will compare passive listening with active conversation in a dyadic communication scenario and manipulate conversation involvement (active vs. passive) and visual representation of interlocutors (real presence vs. telepresence). In Study 2, we will explore auditory processing and interpersonal synchrony between familiar and unfamiliar interlocutors using triadic turn-taking communication scenarios. We will introduce the factor of interlocutor familiarity and investigate the influence of visual interlocutor representation (real presence vs. telepresence) on auditory processing and interpersonal synchrony. Study 3 will investigate auditory processing and interpersonal synchrony in triadic turn-taking situations in the presence of audio-visual distractor scenarios, thereby manipulating communication difficulty. In all three studies, we will use EEG as a non-invasive means of monitoring attention to speech and non-speech signals. In addition, measures of behavioral synchrony between conversing individuals will be developed to predict communication effort.

我们对听觉语音处理和注意力的理解主要源于受控的实验室实验，这些实验使用了简化的人工刺激，这些刺激在复杂性和可预测性方面与现实生活场景不同。虽然视觉线索（如唇读）对听觉语音处理的影响是众所周知的，但在复杂的通信场景（如社交互动）中的视听处理并没有得到太多的关注。现实生活中的交流涉及到带有视觉线索的互动循环，如身体动作、手势、面部表情和眨眼，这些在传统的实验室实验中无法很好地捕捉到，这些实验将倾听者视为被动接收者。我们开发交互式虚拟环境（VE）来推进该领域，因为VE提供了实施和使用可再现的逼真视听场景的机会。我们的虚拟环境联合收割机将可复制性和可再现性的目标与日常通信需求的复杂性相结合。我们的目标是通过开发和评估至少模仿自然通信场景的一些复杂性并将听众作为主动代理的虚拟环境，来加深我们对视听语音处理的理解。在神经水平上，听觉注意到语音的时间动态可以通过脑电图（EEG）进行非侵入性探索。在本项目的第一阶段，在非交互式VE中实施了EEG皮层语音跟踪，并使我们能够客观地评估注意力倾听。具体来说，我们发现，皮层语音跟踪产生有效的结果，但事先记录的故事，在VE。在这个项目的第二阶段，我们将推进我们的虚拟现实，以模拟真实的和远程呈现的条件。这些虚拟环境使研究的生活，互动的通信方案。在研究1中，我们将在一个二元沟通场景中比较被动倾听与主动对话，并操纵对话参与（主动与被动）和对话者的视觉表征（真实的存在与远程呈现）。在研究二中，我们将使用三元话轮转换的交流场景，探讨熟悉和不熟悉的对话者之间的听觉加工和人际同步。我们将引入对话者熟悉度这一因素，并探讨视觉对话者表征（真实的在场与遥现）对听觉加工和人际同步的影响。研究三将探讨视听干扰情景下三元话轮转换情境中的听觉加工和人际同步性，从而控制交际难度。在这三项研究中，我们将使用EEG作为监测对语音和非语音信号的注意力的非侵入性手段。此外，交谈的个人之间的行为同步的措施将被开发来预测沟通的努力。