CHS: Small: Audio-Visual Reconstruction for Immersive Virtualized Reality

在沉浸式虚拟环境中，保持存在感是一大挑战。沉浸感的一个重要方面是不同感官之间的凝聚力，包括视觉和听觉；例如，一个看起来像木头的物体听起来也应该像木头。声音合成可以提高用户与物体互动时的感官凝聚力，但它需要准确的真实世界材料参数。虽然计算机视觉的许多先前工作都集中在获取物体的几何形状和视觉特征上，但由此产生的点云和图像可以帮助更准确地恢复声音合成的音频参数，以及声音渲染和传播的声散射和吸收特性。这项研究的一个假设是，相反，听觉指标也可以帮助确定物体的几何形状，包括孔洞和闭塞，以类似于声纳检测的方式（当然，3D几何形状重建比物体检测更具挑战性）。该项目实现的视听重建将在许多领域产生广泛的影响，包括视障人士的辅助技术、多模式以人为中心的界面、沉浸式电话会议、城市规划声学空间的快速原型设计、结构设计和噪声控制，等等。包括科学进展和软件系统在内的项目成果将通过网站、出版物、研讨会、社区外展和其他专业活动传播。该项目探索了现实世界场景的视听重建的新范例，其中音频线索用于指导物体和材料的分类，以进行3D几何重建。同时，视觉信息可用于初始化和加速声材料参数的识别。将解决的一些主要研究挑战包括音频引导的3D模型重建，用于材料和物体识别的视听神经网络设计，基于学习的大型物理或虚拟空间声学材料分类，以及使用几何和基于波的方法的基于优化的声学材料细化。将对新方法和应用进行基于感知的评估和验证。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Maintaining the sense of presence is a major challenge in immersive virtual environments. An important aspect of immersion is the feeling of cohesiveness between different senses, including the visual and auditory; for example, an object that looks like wood should also sound like wood. Sound synthesis can improve a user's sensory cohesion when interacting with objects, but it requires accurate real-world material parameters. While much prior work in computer vision has focused on acquiring the geometric shape and visual characteristic of objects, the resulting point-clouds and images can assist in more accurate recovery of audio parameters for sound synthesis, along with acoustic scattering and absorption properties for sound rendering and propagation. A hypothesis of this research is that, conversely, auditory metrics can also assist in determining an object's geometry, including holes and occlusions, in a manner analogous to sonar detection (but of course 3D geometry reconstruction is far more challenging than object detection). The audio-visual reconstruction enabled by this project will have broad impact across many domains, including assistive technology for persons who are visually impaired, multimodal human-centric interfaces, immersive teleconferencing, rapid prototyping of acoustic spaces for urban planning, structural design, and noise control, to name just a few. Project outcomes including scientific advances and software systems will be disseminated through websites, publications, workshops, community outreach, and other professional events.This project explores a novel paradigm of audio-visual reconstruction of real-world scenes, where audio cues are used to guide the classification of objects and materials for 3D geometry reconstruction. At the same time, the visual information can be used to initialize and accelerate the identification of acoustic material parameters. Some of the major research challenges that will be addressed include audio-guided 3D model reconstruction, design of audio-visual neural networks for material and object identification, learning-based acoustic material classification of a large physical or virtual space, and optimization-based acoustic material refinement using geometric and wave-based methods. Perceptually-grounded evaluation and validation of the new methods and applications will be performed.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.