Career: Learning Multimodal Representations of the Physical World

触觉和听觉传达了仅凭视觉很难感知的关于世界的物理属性。该项目的目标是赋予机器感知系统在这三种感觉模式之间形成跨模式联系的能力，例如预测物体将如何感觉或从视觉中发出声音的能力。这些跨模式关联也可以通过传感器直接获得，使它们非常适合创建自主系统，学习在没有人工监督的情况下与世界进行物理交互。该项目的综合教育和推广活动也将促进普通观众和多个层次的学生对多模式机器学习的理解。该项目旨在通过视觉、声音和触觉之间的跨模式联系来学习材料属性和微观几何。它通过四个研究推力做到了这一点。首先，它的目标是通过将来自所有模式的观察记录到统一的3D模型中来捕获3D多模式表示，使用估计的视觉几何来从稀疏的观察中获得对触摸和声音的密集估计。其次，它的目标是利用跨模式的视觉监控，在物理交互过程中通过触摸和声音生成物体的时空重建。第三，它的目标是学习捕捉声学属性的材料表示法，以及将这些表示法集成到3D声音合成模型中的方法。最后，它旨在模拟和学习捕获的3D多模式场景中的物理交互。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Touch and hearing convey physical properties about the world that are difficult to perceive from vision alone. The objective of this project is to give machine perception systems the ability to form cross-modal associations between these three sensory modalities, such as the ability to predict how an object will feel or sound from sight. These cross-modal associations can also be obtained directly via sensors, making them well-suited to creating autonomous systems that learn to physically interact with the world without human-provided supervision. The project's integrated education and outreach activities will also advance an understanding of multimodal machine learning for a general audience, and for students at multiple levels.This project aims to learn material properties and microgeometry through cross-modal associations between sight, sound, and touch. It does this through four research thrusts. First, it aims to capture 3D multimodal representations by registering observations from all modalities into a unified 3D model, using estimated visual geometry to obtain dense estimates of touch and sound from sparse observations. Second, it aims to generate space-time reconstructions of objects from touch and sound during physical interaction, using cross-modal visual supervision. Third, it aims to learn material representations that capture acoustic properties, as well as methods that integrate these representations into 3D sound synthesis models. Finally, it aims to simulate and learn physical interactions within captured 3D multimodal scenes.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.