CHS: Small: Audio-Visual Reconstruction for Immersive Virtualized Reality

保持存在感是沉浸式虚拟环境中的主要挑战。沉浸感的一个重要方面是不同感官之间的凝聚力，包括视觉和听觉;例如，一个看起来像木头的物体听起来也应该像木头。声音合成可以提高用户与对象交互时的感官凝聚力，但它需要准确的真实世界材料参数。虽然计算机视觉中的许多先前工作集中于获取对象的几何形状和视觉特性，但是所得到的点云和图像可以帮助更准确地恢复用于声音合成的音频参数，沿着用于声音渲染和传播的声学散射和吸收特性。 这项研究的一个假设是，相反，听觉指标也可以帮助确定对象的几何形状，包括孔和闭塞，以类似于声纳检测的方式（但当然3D几何重建比对象检测更具挑战性）。 该项目实现的视听重建将在许多领域产生广泛影响，包括视障人士的辅助技术，多模态以人为中心的界面，沉浸式电话会议，城市规划声学空间的快速原型，结构设计和噪音控制，仅举几例。 项目成果包括科学进步和软件系统将通过网站、出版物、研讨会、社区外展和其他专业活动进行传播。该项目探索了一种新的真实世界场景视听重建范例，其中音频线索用于指导物体和材料的分类，以进行3D几何重建。同时，视觉信息可用于初始化和加速声学材料参数的识别。将解决的一些主要研究挑战包括音频引导的3D模型重建，用于材料和物体识别的视听神经网络设计，大型物理或虚拟空间的基于学习的声学材料分类，以及基于优化的声学材料细化使用几何和基于波的方法。 该奖项反映了NSF的法定使命，并被认为是值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估的支持。

Maintaining the sense of presence is a major challenge in immersive virtual environments. An important aspect of immersion is the feeling of cohesiveness between different senses, including the visual and auditory; for example, an object that looks like wood should also sound like wood. Sound synthesis can improve a user's sensory cohesion when interacting with objects, but it requires accurate real-world material parameters. While much prior work in computer vision has focused on acquiring the geometric shape and visual characteristic of objects, the resulting point-clouds and images can assist in more accurate recovery of audio parameters for sound synthesis, along with acoustic scattering and absorption properties for sound rendering and propagation. A hypothesis of this research is that, conversely, auditory metrics can also assist in determining an object's geometry, including holes and occlusions, in a manner analogous to sonar detection (but of course 3D geometry reconstruction is far more challenging than object detection). The audio-visual reconstruction enabled by this project will have broad impact across many domains, including assistive technology for persons who are visually impaired, multimodal human-centric interfaces, immersive teleconferencing, rapid prototyping of acoustic spaces for urban planning, structural design, and noise control, to name just a few. Project outcomes including scientific advances and software systems will be disseminated through websites, publications, workshops, community outreach, and other professional events.This project explores a novel paradigm of audio-visual reconstruction of real-world scenes, where audio cues are used to guide the classification of objects and materials for 3D geometry reconstruction. At the same time, the visual information can be used to initialize and accelerate the identification of acoustic material parameters. Some of the major research challenges that will be addressed include audio-guided 3D model reconstruction, design of audio-visual neural networks for material and object identification, learning-based acoustic material classification of a large physical or virtual space, and optimization-based acoustic material refinement using geometric and wave-based methods. Perceptually-grounded evaluation and validation of the new methods and applications will be performed.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.