Career: Learning Multimodal Representations of the Physical World

触觉和听觉传达了仅凭视觉难以感知的世界的物理特性。该项目的目标是使机器感知系统能够在这三种感觉模态之间形成跨模态关联，例如预测物体的感觉或声音的能力。这些跨模态关联也可以通过传感器直接获得，这使得它们非常适合创建自主系统，这些系统可以在没有人类监督的情况下学习与世界进行物理交互。该项目的综合教育和推广活动也将促进普通观众和多层次学生对多模态机器学习的理解。该项目旨在通过视觉，声音和触觉之间的跨模态关联来学习材料属性和微观几何。它通过四个研究重点来实现这一点。首先，它旨在通过将来自所有模态的观测记录到统一的3D模型中来捕获3D多模态表示，使用估计的视觉几何形状从稀疏观测中获得对触摸和声音的密集估计。其次，它的目的是在物理交互过程中，使用跨模态视觉监督，从触摸和声音中生成对象的时空重建。第三，它旨在学习捕获声学特性的材料表示，以及将这些表示集成到3D声音合成模型中的方法。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Touch and hearing convey physical properties about the world that are difficult to perceive from vision alone. The objective of this project is to give machine perception systems the ability to form cross-modal associations between these three sensory modalities, such as the ability to predict how an object will feel or sound from sight. These cross-modal associations can also be obtained directly via sensors, making them well-suited to creating autonomous systems that learn to physically interact with the world without human-provided supervision. The project's integrated education and outreach activities will also advance an understanding of multimodal machine learning for a general audience, and for students at multiple levels.This project aims to learn material properties and microgeometry through cross-modal associations between sight, sound, and touch. It does this through four research thrusts. First, it aims to capture 3D multimodal representations by registering observations from all modalities into a unified 3D model, using estimated visual geometry to obtain dense estimates of touch and sound from sparse observations. Second, it aims to generate space-time reconstructions of objects from touch and sound during physical interaction, using cross-modal visual supervision. Third, it aims to learn material representations that capture acoustic properties, as well as methods that integrate these representations into 3D sound synthesis models. Finally, it aims to simulate and learn physical interactions within captured 3D multimodal scenes.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.