Examining attention, memory performance, and listening effort (exAMPLE): Understanding listeners' cognitive performances in complex audiovisual communication settings with embodied conversational agents

面对面交流是最普遍的口头信息交流形式。然而，在复杂和动态的环境中，如开放式办公室和忙碌的酒吧，听众的多说话者的谈话遇到独特的挑战，由于声学复杂，刺激丰富，和动态的性质，这些多感官设置。复杂的视听呈现对多人谈话的理解和记忆的影响，以及在听的时候记笔记的影响，在很大程度上还没有被探索。这项研究计划旨在调查虚拟现实（VR）中真实听力环境中的视听变化，并评估其对听众注意力，语音清晰度，听力理解以及对话内容和其他谈话者相关方面的记忆的影响（例如，谁说的）。在我们的前一个项目的基础上，这项研究从简化到复杂的视听沉浸式环境，在听觉和视觉刺激方面都是丰富和充满活力的。认知表现评估的范围从理解和回忆会话事实扩展到包括指导性笔记和检索其他会话方面，包括注意力分配和指导。为了实现这一目标，我们将开发沉浸式和交互式视听虚拟环境，参与者与两个或更多的谈话者进行对话，代表具体的会话代理。随后，将测试参与者对对话内容和其他对话相关细节的回忆。我们将从我们最初的项目中扩展和调整听到的文本回忆（HTR）范式，以包括笔记（NT）和关于谈话者相关（TR）方面的问题，最终分别导致HTR-NT和HTR-TR任务。通过探索视听VR环境特征对上述认知表现的影响，本研究将为创建逼真和沉浸式的言语社交互动提供指导，从而促进正在进行的基于VR的认知研究。

Face-to-face communication is the most prevalent form of verbal information exchange. However, in complex and dynamic environments such as open-plan offices and busy bars, listeners of multi-talker conversations encounter unique challenges due to the acoustically complex, stimulus-rich, and dynamic nature of these multisensory settings. The effects of complex audiovisual presentations on comprehension and memory of multi-talker conversations, as well as the influence of note-taking while listening, remain largely unexplored. This research proposal aims to investigate audiovisual variations in realistic listening settings in Virtual Reality (VR) and assess their impact on listener attention, speech intelligibility, listening comprehension, as well as memory for conversational content and other talker-related aspects (e.g., who said what). Building upon our predecessor project, this study advances from simplified to intricate audiovisual immersive environments that are rich and vibrant in both auditory and visual stimuli. The scope of cognitive performance assessment expands from understanding and recalling conversational facts to encompass guided note-taking and the retrieval of additional conversational aspects, including attention allocation and guidance. To achieve this, we will develop immersive and interactive audiovisual virtual environments in which participants engage in conversations with two or more talkers, represented as embodied conversational agents. Subsequently, participants will be tested on their recall of conversational content and other conversation-related details. We will extend and adapt the Heard Text Recall (HTR) paradigm from our initial project to include note-taking (NT) and questions about talker-related (TR) aspects, finally resulting in the HTR-NT and HTR-TR tasks, respectively. By exploring the effects of audiovisual VR environment characteristics on the aforementioned cognitive performances, this research will provide guidelines for creating realistic and immersive verbal social interactions, thus facilitating ongoing VR-based cognitive research.