Immersive Audio-Visual 3D Scene Reproduction Using a Single 360 Camera

新冠肺炎疫情改变了我们的生活方式，对远程沟通和体验提出了很高的要求。许多组织不得不建立带有视频会议平台的远程工作系统。然而，由于缺乏目光接触、凝视意识和空间音频同步，目前的视频会议系统不能满足远程协作的基本要求。将真实空间以视听三维模型的形式再现，使用户可以在真实环境中远程体验实时交互，因此可以广泛应用于医疗保健、远程会议、教育、娱乐等各种应用。该项目的目标是开发一种简单实用的解决方案来估计一般场景的几何结构和声学特性，从而使空间音频适应环境和听众的位置，从而提供沉浸式的场景渲染，以改善用户体验。现有的3D场景再现系统存在两个问题。(i)分别研究了视听系统。计算机视觉的研究主要集中在提高场景重建的视觉方面。在沉浸式显示器中，如VR系统，如果声音与视觉线索不匹配，用户就不会认为这种体验是“真实的”。另一方面，音频研究一直只使用音频传感器来测量声学特性，而没有考虑与视觉传感器的互补效应。（ii）目前用于3D场景再现的捕捉和记录系统需要过于侵入性的设置和专业的过程，用户无法在私人场所部署。激光雷达传感器价格昂贵，需要很长的扫描时间。透视图像需要大量的照片来覆盖整个场景。本研究的目的是开发一个端到端的视听3D场景再现管道，使用消费者360（全景）相机的单个镜头。为了让普通用户在自己的私人空间中轻松访问系统，在后端应包含使用计算机视觉和人工智能算法的自动解决方案。将开发一种用于捕获环境的语义场景重建和声学特性预测的深度神经网络（DNN）。这个过程包括对来自相机的不可见区域的推断。脉冲响应（IRs）表征环境的声学属性，允许用任何声源再现空间的声学。它还允许通过消除录制声音中的声学效果来提取原始（干）声音，以便该源可以在具有不同声学效果的新环境中重新呈现。本文将研究一种简单有效的方法，从捕获的单张360度照片中估计声红外光谱。这些语义场景数据用于为用户提供身临其境的视听体验。将考虑两种类型的显示场景：个性化显示系统，如带有耳机的VR头显和带有扬声器的公共显示系统（如电视或投影仪）。将开发使用单个360摄像机的实时3D人体姿势跟踪，以便在用户位置准确渲染3D视听场景。使用扬声器向听众传递双耳声音是一项具有挑战性的任务。音频波束形成技术将与多个扬声器的人体姿态跟踪相结合，将与项目伙伴在音频处理方面进行合作研究。由此产生的系统将对虚拟现实和多媒体系统的创新产生重大影响，并为它们的部署开辟新的和有趣的应用。该奖项应为PI建立和领导一个具有独特研究方向的小组提供基础，该研究方向应与国家优先事项保持一致，并将解决重大的长期研究挑战。

The COVID-19 pandemic has changed our lifestyle and caused high demand for remote communication and experience. Many organizations have had to set up remote work systems with video conferencing platforms. However, current video conferencing systems do not meet basic requirements for remote collaboration due to the lack of eye contact, gaze awareness and spatial audio synchronisation. Reproduction of a real space as an audio-visual 3D model allows users to remotely experience real-time interaction in real environments, thus it can be widely utilised in various applications such as healthcare, teleconferencing, education, entertainments, etc. The goal of this project is to develop a simple and practical solution to estimate geometrical structure and acoustic properties of general scenes allowing spatial audio to be adapted to the environment and listener location to give an immersive rendering of the scene to improve user experience.Existing 3D scene reproduction systems have two problems. (i) Audio and vision systems have been researched separately. Computer vision research has mainly focused on improving the visual side of scene reconstruction. In an immersive display, such as a VR system, the experience is not perceived as "realistic" by users if sound is not matched with the visual cues. On the other hand, audio researches have been using only audio sensors to measure acoustic properties without considering the complementary effect with visual sensors. (ii) Current capture and recording systems for 3D scene reproduction require too invasive set up and professional process to be deployed by users in their private places. A LiDAR sensor is expensive and requires long scanning time. Perspective images require large number of photos to cover the whole scene. The objective of this research is to develop an end-to-end audio-visual 3D scene reproduction pipeline using a single shot from a consumer 360 (panoramic) camera. In order to make the system easily accessible by common users in their own private spaces, automatic solution using computer vision and artificial intelligence algorithms should be included in the back-end. A deep neural network (DNN) jointly trained for semantic scene reconstruction and acoustic property prediction for the captured environments will be developed. This process includes inference for invisible regions from the camera. Impulse Responses (IRs) characterising acoustic attributes of an environment allow to reproduce the acoustics of the space with any sound sources. It also allows to extract the original (dry) sound by eliminating acoustic effects from recorded sound so that this source can be re-rendered in new environments with different acoustic effects. A simple and efficient method to estimate acoustic IRs from the captured single 360 photo will be investigated. This semantic scene data is used to provide immersive audio-visual experience to users. Two types of display scenarios will be considered: personalised display system such as a VR headset with headphones and communal display system (e.g., TV or projector) with loudspeakers. Real-time 3D human pose tracking using a single 360 camera will be developed to accurately render 3D audio-visual scene at the locations of users. Delivering binaural sound to listeners using loudspeakers is a challenging task. Audio beam-forming techniques aligned with human-pose tracking for multiple loudspeakers will be investigated in collaboration with the project partners in audio processing. The resulting system would have a significant impact on innovation of VR and multimedia systems, and open up new and interesting applications for their deployment. This award should provide the foundation for the PI to establish and lead a group with a unique research direction which is aligned with national priorities and will address a major long-term research challenge.