Spatial Sound Generation and Perceptual-Based Rendering for Virtual Environments and Serious Games

虚拟环境中的空间声音，如虚拟模拟、严肃游戏或视频游戏，允许用户感知声源在三维空间中任意位置的位置，当适当地再现时，它可以提供一种非常逼真的感觉，即远程沉浸在人、乐器和环境声音的存在中。空间声音可以增加一层新的真实感，有助于更好的用户体验和改进任务性能，同时传达使用其他方式（如视觉）难以传达的信息。然而，在沉浸式虚拟环境中，空间声音往往被忽视，传统上，强调的是视觉场景（即“计算机图形”）。也就是说，动态、交互式和沉浸式虚拟环境的空间声音生成是困难的，而且计算成本很高，除了使用有限的琐碎环境。**在拟议的研究计划范围内，将开发一个虚拟声音渲染引擎（VSRE），该引擎将允许以简单有效的方式将空间声音整合到交互式动态虚拟环境中。将通过一系列定量和定性技术对VSRE进行严格验证，验证结果将提供给研究界，从而允许其他人将VSRE与自己的空间声音系统进行比较。VSRE还将被开发为允许对声学环境和音频参数（音质）进行简单的操作。然后，它将被整合到现有的严肃游戏中，从而形成一个“研究试验台”，用于通过与人类参与者进行的一系列实验，测试不同的声学环境和音频参数可能对虚拟环境中的沉浸感、参与度和视觉保真度感知产生的影响。最后，将开发一种方法，该方法将允许虚拟环境的各个方面，包括音频和视觉保真度水平以及背景声音，通过用户将在使用虚拟环境之前完成的简短校准过程专门为用户量身定制。这将带来更吸引人的用户体验。**拟议的研究计划将导致i)开发一种新颖而独特的虚拟声音渲染引擎，该引擎将允许以交互速率将空间声音包含在虚拟环境中；ii)更好地理解空间声音及其与视觉线索的交互在交互式虚拟环境中所起的作用，以及如何使用空间声音来克服这些环境中固有的潜在限制。Iii)一种新颖的校准方法，可以专门为每个用户量身定制虚拟环境，iv)为多模态虚拟环境（特别是虚拟模拟和严肃游戏）的设计师/开发人员提供一套“最佳实践”（指南），以最大限度地提高这种环境中的参与度、沉浸感和整体用户体验。总的来说，拟议的研究计划将产生更有效的虚拟模拟和严肃游戏，最终以安全，高效和经济的方式改善特定劳动力（外科医生/医疗保健专业人员，警察等）的培训，确保他们更好地准备执行预期的任务。

Spatial sound within a virtual environment such as a virtual simulation, serious game, or video game, allows users to perceive the position of a sound source at an arbitrary position in three-dimensional space and when properly reproduced, it can deliver a very life-like sense of being remotely immersed in the presence of people, musical instruments, and environmental sounds. Spatial sound can add a new layer of realism, contributing to a greater user experience and improved task performance, while conveying information that would otherwise be difficult to convey using other modalities (e.g., vision). However, spatial sound is often overlooked in immersive virtual environments where traditionally, emphasis has been placed on the visual scene (i.e., the "computer graphics") instead. That being said, spatial sound generation for dynamic, interactive, and immersive virtual environments is difficult and computationally expensive except for trivial environments which are of limited use. **Within the scope of the proposed research program, a virtual sound rendering engine (VSRE) that will allow for the incorporation of spatial sound within an interactive, dynamic virtual environments in a simple and efficient manner will be developed. The VSRE will be rigorously validated through a series of quantitative and qualitative techniques and the validation results will be made available to the research community thus allowing others to compare the VSRE with their own spatial sound systems. The VSRE will also be developed to allow for the simple manipulation of the acoustic environment and audio parameters (sound quality). It will then be incorporated into an existing serious game thus forming a "research test-bed" which will be used to test what effect varying the acoustic environment and audio parameters may have on immersion, engagement, and visual fidelity perception within a virtual environment via a series of experiments that will be conducted with human participants. Finally, a method will be developed that will allow various aspects of a virtual environment including levels of audio and visual fidelity, and background sound, to be tailored specifically to the user via a brief calibration process that the user will complete prior to using the virtual environment. This will result in a more engaging user experience. **The proposed research program will lead to i) the development of a novel and unique virtual sound rendering engine that will allow for the inclusion of spatial sound within virtual environments at interactive rates, ii) a greater understanding of the role that spatial sound and its interaction with the visual cues play in interactive virtual environments and how spatial sound can be used to overcome potential limitations inherent in these environments, iii) a novel calibration method that will tailor the virtual environment specifically to each user, and iv)a set of "best practices" (guidelines) for designers/developers of multi-modal virtual environments (particularly virtual simulations, and serious games), to maximize engagement, immersion and overall user experience within such environments. Collectively, the proposed research program will result in more effective virtual simulations and serious games ultimately leading to improved training of a specific workforce (surgeons/healthcare professionals, police officers, etc.) in a safe, efficient, and cost-effective manner, ensuring they are better prepared to carry out their intended tasks.