Career: Learning Multimodal Representations of the Physical World

触觉和听觉传达了世界的物理特性，这些特性仅凭视觉是很难感知的。该项目的目标是使机器感知系统能够在这三种感官模式之间形成跨模态关联，例如能够从视觉上预测物体的感觉或声音。这些跨模式关联也可以直接通过传感器获得，这使得它们非常适合创建自主系统，这些系统可以在没有人为监督的情况下学习与世界进行物理交互。该项目的综合教育和推广活动也将为普通受众和不同层次的学生提高对多模式机器学习的理解。该项目旨在通过视觉、声音和触觉之间的跨模态关联来学习材料的特性和微观几何。它通过四个研究重点来做到这一点。首先，它旨在通过将所有模态的观测值注册到统一的3D模型中来捕获3D多模态表示，使用估计的视觉几何来从稀疏观测中获得触摸和声音的密集估计。其次，利用跨模态视觉监督，通过触觉和声音生成物体在物理交互过程中的时空重构。第三，它旨在学习捕捉声学特性的材料表征，以及将这些表征整合到3D声音合成模型中的方法。最后，它的目的是模拟和学习捕获的3D多模态场景中的物理相互作用。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Touch and hearing convey physical properties about the world that are difficult to perceive from vision alone. The objective of this project is to give machine perception systems the ability to form cross-modal associations between these three sensory modalities, such as the ability to predict how an object will feel or sound from sight. These cross-modal associations can also be obtained directly via sensors, making them well-suited to creating autonomous systems that learn to physically interact with the world without human-provided supervision. The project's integrated education and outreach activities will also advance an understanding of multimodal machine learning for a general audience, and for students at multiple levels.This project aims to learn material properties and microgeometry through cross-modal associations between sight, sound, and touch. It does this through four research thrusts. First, it aims to capture 3D multimodal representations by registering observations from all modalities into a unified 3D model, using estimated visual geometry to obtain dense estimates of touch and sound from sparse observations. Second, it aims to generate space-time reconstructions of objects from touch and sound during physical interaction, using cross-modal visual supervision. Third, it aims to learn material representations that capture acoustic properties, as well as methods that integrate these representations into 3D sound synthesis models. Finally, it aims to simulate and learn physical interactions within captured 3D multimodal scenes.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.