Influence of audio rendering in virtual environments on realism, presence, and socio-cognitive processing

交互式虚拟环境（IVEs）为研究复杂场景中的人类感知和社会认知过程提供了独特的新可能性。在人工智能中，生态上有效的环境条件可以与高度的实验控制相结合。虽然研究人员经常将音频作为一种模态，但将高度逼真的音频整合在一起仍然是一个例外。该项目的总体目标是确定双耳音频渲染的需求，以实现虚拟社会场景的高度逼真的视听呈现，并表征音频渲染对（社会）存在和社会认知过程（如情感、同理心和（社会）焦虑）的影响。在第一个资助阶段，重点是实现尽可能高的现实主义，在第二个资助阶段，我们将集中在自我报告、行为和神经水平（通过脑电图测量）上描述虚拟社会互动场景中自上而下效应和刺激相关的自下而上效应之间的相互作用。通过考虑和分析期望、情绪、熟悉程度、社会压力、存在以及高阶社会认知（例如，共情），这种互动将被调查，这些虚拟社会场景具有不同程度的复杂性，涉及这些场景中的双耳呈现：在第一阶段开发的（高真实感）渲染是起点，它将扩展到多个语音和非语音礼堂相关的来源，以及更高级的自己的声音渲染，以实现接近现实的互动社交场景。将研究习惯化和训练，以及头部跟踪的替代方法，以促进视听人工智能的使用，并促进进一步的临床应用，如治疗社交焦虑症。总的来说，这个项目将有助于更好地理解心理和生理过程及其功能相关性，以提供真实环境的合理表征。

Interactive Virtual Environments (IVEs) provide unique new possibilities to study human perception and socio-cognitive processing in complex scenes. In IVEs, ecologically valid environmental conditions can be combined with a high degree of experimental control. Although researchers often included audio as a modality, the integration of highly realistic audio is still an exception. The overarching goal of this project is to identify the requirements regarding binaural audio rendering to achieve highly realistic audiovisual presentations of virtual social scenes and to characterize the effects of the audio rendering on (social) presence and socio-cognitive processes such as emotion, empathy, and (social) anxiety. Whereas in the first funding phase, the focus was on achieving the highest possible realism, in the second funding phase we will concentrate on characterizing the interaction between top-down effects and stimulus-related bottom-up effects in virtual social interaction scenarios on a self-report, behavioral, and on a neural level (measured via EEG). This interaction will be investigated by considering and analyzing expectations, emotion, familiarization, social stress, and presence as well as higher-order social cognition (e.g., empathy) in virtual social scenarios with different degrees of complexity regarding the binaural rendering in these scenarios: the (high realism) rendering developed in phase 1 is the starting point, which will be extended to multiple speech and non-speech auditorium-related sources as well as to a more advanced own-voice renderings in order to enable close-to-real-life interactive social scenes. Habituation and training, as well as alternatives to head-tracking, will be investigated to facilitate the use of audiovisual IVEs and to enable further clinical applications such as treating social anxiety disorders. Overall this project will contribute to better understand psychological and physiological processes and their functional relevance to provide plausible representations of real-life environments.