Examining attention, memory performance, and listening effort (exAMPLE): Understanding listeners' cognitive performances in complex audiovisual communication settings with embodied conversational agents

面对面的交流是最普遍的口头信息交流形式。然而，在复杂和动态的环境中，如开放式办公室和繁忙的酒吧，由于这些多感官环境的声学复杂性，刺激丰富和动态性，多谈话的听众遇到了独特的挑战。复杂的视听演示对多说话者对话的理解和记忆的影响，以及在听时做笔记的影响，在很大程度上仍未被探索。本研究计划旨在调查虚拟现实（VR）中真实听力设置中的视听变化，并评估它们对听者注意力、语音清晰度、听力理解、会话内容记忆和其他与谈话相关的方面（例如，谁说了什么）的影响。在我们的前一个项目的基础上，本研究从简化到复杂的视听沉浸式环境，在听觉和视觉刺激方面都丰富而充满活力。认知表现评估的范围从理解和回忆会话事实扩展到包括引导笔记和检索额外的会话方面，包括注意力分配和指导。为了实现这一目标，我们将开发沉浸式和交互式视听虚拟环境，参与者与两个或更多的谈话者进行对话，代表为具体化的对话代理。随后，参与者将被测试他们对对话内容和其他对话相关细节的回忆。我们将从最初的项目中扩展和调整听到的文本回忆（HTR）范式，包括记笔记（NT）和关于说话人相关（TR）方面的问题，最终分别产生HTR-NT和HTR-TR任务。通过探索视听VR环境特征对上述认知表现的影响，本研究将为创造逼真的沉浸式语言社会互动提供指导，从而促进基于VR的认知研究的进行。

Face-to-face communication is the most prevalent form of verbal information exchange. However, in complex and dynamic environments such as open-plan offices and busy bars, listeners of multi-talker conversations encounter unique challenges due to the acoustically complex, stimulus-rich, and dynamic nature of these multisensory settings. The effects of complex audiovisual presentations on comprehension and memory of multi-talker conversations, as well as the influence of note-taking while listening, remain largely unexplored. This research proposal aims to investigate audiovisual variations in realistic listening settings in Virtual Reality (VR) and assess their impact on listener attention, speech intelligibility, listening comprehension, as well as memory for conversational content and other talker-related aspects (e.g., who said what). Building upon our predecessor project, this study advances from simplified to intricate audiovisual immersive environments that are rich and vibrant in both auditory and visual stimuli. The scope of cognitive performance assessment expands from understanding and recalling conversational facts to encompass guided note-taking and the retrieval of additional conversational aspects, including attention allocation and guidance. To achieve this, we will develop immersive and interactive audiovisual virtual environments in which participants engage in conversations with two or more talkers, represented as embodied conversational agents. Subsequently, participants will be tested on their recall of conversational content and other conversation-related details. We will extend and adapt the Heard Text Recall (HTR) paradigm from our initial project to include note-taking (NT) and questions about talker-related (TR) aspects, finally resulting in the HTR-NT and HTR-TR tasks, respectively. By exploring the effects of audiovisual VR environment characteristics on the aforementioned cognitive performances, this research will provide guidelines for creating realistic and immersive verbal social interactions, thus facilitating ongoing VR-based cognitive research.