Spatial Sound Generation and Perceptual-Based Rendering for Virtual Environments and Serious Games

虚拟环境中的空间声音，如虚拟仿真，严肃游戏或视频游戏，允许用户感知声源在三维空间中任意位置的位置，并且当正确再现时，它可以提供远程沉浸在人，乐器和环境声音中的非常逼真的感觉。 空间声音可以添加新的现实主义层，有助于更好的用户体验和改进的任务性能，同时传达使用其他模态难以传达的信息（例如，视觉）。 然而，空间声音在沉浸式虚拟环境中经常被忽视，在沉浸式虚拟环境中，传统上，重点放在视觉场景上（即，“计算机图形”）。也就是说，用于动态、交互式和沉浸式虚拟环境的空间声音生成是困难的并且计算上昂贵，除了用途有限的琐碎环境之外。 ** 在拟议的研究计划范围内，将开发一个虚拟声音渲染引擎（VSRE），该引擎将允许以简单有效的方式将空间声音纳入交互式动态虚拟环境。 VSRE将通过一系列定量和定性技术进行严格验证，验证结果将提供给研究界，从而允许其他人将VSRE与他们自己的空间音响系统进行比较。 还将开发VSRE，以便对声学环境和音频参数（音质）进行简单操作。 然后，它将被纳入现有的严肃游戏中，从而形成一个“研究测试台”，该测试台将用于通过一系列将与人类参与者进行的实验来测试改变声学环境和音频参数可能对虚拟环境中的沉浸感、参与度和视觉保真度感知产生什么影响。 最后，将开发一种方法，该方法将允许虚拟环境的各个方面，包括音频和视觉保真度的水平以及背景声音，通过用户在使用虚拟环境之前将完成的简短校准过程来专门针对用户定制。这将带来更具吸引力的用户体验。** 拟议的研究计划将导致i）开发一种新颖独特的虚拟声音渲染引擎，该引擎将允许以交互速率在虚拟环境中包含空间声音，ii）第二阶段更好地理解空间声音及其与视觉提示在交互式虚拟环境中的作用，以及如何使用空间声音来克服这些环境中固有的潜在限制iii）一种新颖的校准方法，其将专门针对每个用户定制虚拟环境，以及iv）一组用于多模态虚拟环境（特别是虚拟模拟和严肃游戏）的设计者/开发者的“最佳实践”（指南），以最大化在这样的环境内的参与、沉浸和整体用户体验。总的来说，拟议的研究计划将导致更有效的虚拟模拟和严肃的游戏，最终导致改善特定劳动力（外科医生/医疗保健专业人员，警察等）的培训。以安全、高效和具有成本效益的方式，确保他们更好地准备执行预期任务。

Spatial sound within a virtual environment such as a virtual simulation, serious game, or video game, allows users to perceive the position of a sound source at an arbitrary position in three-dimensional space and when properly reproduced, it can deliver a very life-like sense of being remotely immersed in the presence of people, musical instruments, and environmental sounds. Spatial sound can add a new layer of realism, contributing to a greater user experience and improved task performance, while conveying information that would otherwise be difficult to convey using other modalities (e.g., vision). However, spatial sound is often overlooked in immersive virtual environments where traditionally, emphasis has been placed on the visual scene (i.e., the "computer graphics") instead. That being said, spatial sound generation for dynamic, interactive, and immersive virtual environments is difficult and computationally expensive except for trivial environments which are of limited use. **Within the scope of the proposed research program, a virtual sound rendering engine (VSRE) that will allow for the incorporation of spatial sound within an interactive, dynamic virtual environments in a simple and efficient manner will be developed. The VSRE will be rigorously validated through a series of quantitative and qualitative techniques and the validation results will be made available to the research community thus allowing others to compare the VSRE with their own spatial sound systems. The VSRE will also be developed to allow for the simple manipulation of the acoustic environment and audio parameters (sound quality). It will then be incorporated into an existing serious game thus forming a "research test-bed" which will be used to test what effect varying the acoustic environment and audio parameters may have on immersion, engagement, and visual fidelity perception within a virtual environment via a series of experiments that will be conducted with human participants. Finally, a method will be developed that will allow various aspects of a virtual environment including levels of audio and visual fidelity, and background sound, to be tailored specifically to the user via a brief calibration process that the user will complete prior to using the virtual environment. This will result in a more engaging user experience. **The proposed research program will lead to i) the development of a novel and unique virtual sound rendering engine that will allow for the inclusion of spatial sound within virtual environments at interactive rates, ii) a greater understanding of the role that spatial sound and its interaction with the visual cues play in interactive virtual environments and how spatial sound can be used to overcome potential limitations inherent in these environments, iii) a novel calibration method that will tailor the virtual environment specifically to each user, and iv)a set of "best practices" (guidelines) for designers/developers of multi-modal virtual environments (particularly virtual simulations, and serious games), to maximize engagement, immersion and overall user experience within such environments. Collectively, the proposed research program will result in more effective virtual simulations and serious games ultimately leading to improved training of a specific workforce (surgeons/healthcare professionals, police officers, etc.) in a safe, efficient, and cost-effective manner, ensuring they are better prepared to carry out their intended tasks.