CHS: Small: Audio-Visual Reconstruction for Immersive Virtualized Reality

在身临其境的虚拟环境中，保持临场感是一项重大挑战。沉浸的一个重要方面是不同感官之间的凝聚感，包括视觉和听觉；例如，看起来像木头的物体也应该听起来像木头。声音合成可以提高用户在与物体互动时的感觉凝聚力，但它需要准确的真实世界材料参数。虽然计算机视觉中的许多先前的工作都集中在获取对象的几何形状和视觉特征上，但是所得到的点云和图像可以帮助更准确地恢复用于声音合成的音频参数，以及用于声音呈现和传播的声学散射和吸收特性。这项研究的一个假设是，反过来，听觉指标也可以帮助确定对象的几何形状，包括孔洞和遮挡，以类似于声纳检测的方式(但当然，3D几何形状重建比对象检测更具挑战性)。该项目实现的视听重建将在许多领域产生广泛影响，包括视障人士的辅助技术、以人为中心的多模式界面、身临其境的电话会议、用于城市规划、结构设计和噪音控制的声学空间的快速原型制作等。包括科学进展和软件系统在内的项目成果将通过网站、出版物、研讨会、社区推广和其他专业活动进行传播。该项目探索了一种新的视听重建真实世界场景的范例，其中使用音频线索来指导用于3D几何重建的对象和材料的分类。同时，可以利用视觉信息对声学材料参数进行初始化和加速识别。将解决的一些主要研究挑战包括音频引导的3D模型重建，用于材料和对象识别的视听神经网络的设计，基于学习的大型物理或虚拟空间的声学材料分类，以及使用几何和基于波的方法基于优化的声学材料精细化。将对新方法和应用程序进行基于感知的评估和验证。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Maintaining the sense of presence is a major challenge in immersive virtual environments. An important aspect of immersion is the feeling of cohesiveness between different senses, including the visual and auditory; for example, an object that looks like wood should also sound like wood. Sound synthesis can improve a user's sensory cohesion when interacting with objects, but it requires accurate real-world material parameters. While much prior work in computer vision has focused on acquiring the geometric shape and visual characteristic of objects, the resulting point-clouds and images can assist in more accurate recovery of audio parameters for sound synthesis, along with acoustic scattering and absorption properties for sound rendering and propagation. A hypothesis of this research is that, conversely, auditory metrics can also assist in determining an object's geometry, including holes and occlusions, in a manner analogous to sonar detection (but of course 3D geometry reconstruction is far more challenging than object detection). The audio-visual reconstruction enabled by this project will have broad impact across many domains, including assistive technology for persons who are visually impaired, multimodal human-centric interfaces, immersive teleconferencing, rapid prototyping of acoustic spaces for urban planning, structural design, and noise control, to name just a few. Project outcomes including scientific advances and software systems will be disseminated through websites, publications, workshops, community outreach, and other professional events.This project explores a novel paradigm of audio-visual reconstruction of real-world scenes, where audio cues are used to guide the classification of objects and materials for 3D geometry reconstruction. At the same time, the visual information can be used to initialize and accelerate the identification of acoustic material parameters. Some of the major research challenges that will be addressed include audio-guided 3D model reconstruction, design of audio-visual neural networks for material and object identification, learning-based acoustic material classification of a large physical or virtual space, and optimization-based acoustic material refinement using geometric and wave-based methods. Perceptually-grounded evaluation and validation of the new methods and applications will be performed.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.