Challenges in Immersive Audio Technology

沉浸式技术不仅会改变我们的交流和娱乐体验方式，还会改变我们对实体世界的体验，从商店到博物馆，从汽车到教室。这种转变主要是由视觉技术的空前进步推动的，它使用户能够进入另一种视觉现实。然而，在音频领域，为了实现引人注目的沉浸式体验，一群听众可以走进一个场景，感觉被传送到另一个现实，享受无缝的共享体验，而不需要耳机、头部跟踪、个性化或校准，还需要克服一些长期存在的基本挑战。第一个关键挑战是向多个听众提供身临其境的音频体验。音频技术的最新进展开始成功地产生高质量的沉浸式音频体验。然而，这些在实践中仅限于单个听者，通过耳机或基于少量扬声器的系统使用串音消除或波束形成来呈现适当的信号。无论在家庭环境、博物馆、画廊和其他公共场所（5-20人）还是在电影院和剧院礼堂（20-100人）中，将“3D声音”高效地传递给多名听众，在技术上仍然存在根本性的挑战。原则上，共享的听觉体验可以使用基于物理的方法产生，如波场合成或高阶双声，但即使是适度大小的最佳点也需要大量的通道。CIAT的目标是通过开发一个原则性的可扩展和可重构的框架来改变艺术状态，该框架仅用于捕获和再现感知相关的信息，从而通过实际可行的系统实现沉浸式音频体验的质量。第二个关键挑战是环境声学的实时计算，需要将听众转移到另一个现实中，允许他们与环境和声源互动。这与沉浸式音频内容是合成而不是录制的应用程序和基于对象的音频有关。声事件的声场由直接波前、早期反射和高阶反射组成。一种被传送到事件发生的环境的令人信服的体验需要渲染这些反射，而这些反射不能全部实时计算。在真实感至关重要的应用中，例如扩展现实（XR）和某种程度上的游戏，环境的脉冲响应通常只在几个位置计算，预先限制反射的数量和到达的方向，然后与源声音卷积以实现所谓的高质量混响。尽管如此，脉冲响应和卷积的计算可能需要GPU实现，并在质量和复杂性之间以及CPU和GPU计算之间进行仔细的手动平衡。CIAT的目标是在环境建模中提供一种范式转变，使数字高效、无缝、高质量的实时环境模拟成为可能。通过解决这些挑战，CIAT将为新兴的XR应用程序创建和提供共享的交互式沉浸式音频体验，同时在传统媒体的沉浸式音频质量方面取得进步。特别是，高质量环境声学的高效实时合成对于XR和基于对象的音频（包括流媒体和广播）来说都是必不可少的。在广播、电影、音乐活动和视听装置等传统应用中，向多个听众传送3D音景也是一个主要的未解决问题。

Immersive technologies will transform not only how we communicate and experience entertainment, but also our experience of the physical world, from shops to museums, cars to classrooms. This transformation has been driven primarily by an unprecedented progress in visual technologies, which enable transporting users to an alternate visual reality. In the domain of audio, there are however long-standing fundamental challenges that need to be overcome to enable striking immersive experiences in which a group of listeners can just walk into a scene and feel transported to an alternate reality to enjoy a seamless shared experience without the need for headphones, head-tracking, personalisation or calibration.The first key challenge is the delivery of immersive audio experiences to multiple listeners. Recent advances in audio technology are beginning to succeed in generating high quality immersive audio experiences. However, these are restricted in practice to individual listeners, with appropriate signals presented either via headphones, or via systems based on a modest number of loudspeakers using either cross-talk cancellation or beamforming. There remains a fundamental challenge in the technologically efficient delivery of "3D sound" to multiple listeners, either in small numbers (2-5) in a home environment, in museums, galleries and other public spaces (5-20) or in cinema and theatre auditoria (20-100). In principle, shared auditory experiences can be generated using physics-based methods such as wavefield synthesis or higher order ambisonics, but a sweet spot of even a modest size requires a prohibitive number of channels. CIAT aims to transform state of the art by developing a principled scalable and reconfigurable framework for capturing and reproducing only perceptually relevant information, thus leading to a step advance in the quality of immersive audio experiences achievable by practically viable systems.The second key challenge is the real-time computation of environment acoustics needed to transport listeners to alternate reality, allowing them to interact with the environment and sound sources in it. This is pertinent to applications where immersive audio content is synthesised rather than recorded and to object-based audio in general. The sound field of an acoustic event consists of direct wavefront, followed by early and higher-order reflections. A convincing experience of being transported to the environment where the event takes place requires the rendering of these reflections, which cannot all be computed in real time. In applications where the sense of realism is critical, e.g. extended reality (XR) and to some extent gaming, impulse responses of the environment are typically computed only at several locations, with preset limits on the number reflections and directions of arrival, and then convolved with source sounds to achieve what is referred to as high-quality reverberation. Still, the computation of impulse responses and convolution may require GPU implementation and careful hands-on balancing between quality and complexity, and between CPU and GPU computation. CIAT aims to deliver a paradigm shift in environment modelling that will enable numerically efficient seamless high quality environment simulation in real time.By addressing these challenges, CIAT will enable creation and delivery of shared interactive immersive audio experiences for emerging XR applications, whilst making a step advance in the quality of immersive audio in traditional media. In particular, efficient real-time synthesis of high quality environment acoustics is essential for both XR and object-based audio in general, including streaming and broadcasting. Delivery of 3D soundscapes to multiple listeners is a major unresolved problem in traditional applications too, including broadcasting, cinema, music events, and audio-visual installations.