Collaborative Research: RI: Medium: Learning Compositional Implicit Representations for 3D Scene Understanding

场景理解系统获取图像或视频等视觉输入，并根据3D结构、汽车和人等对象以及其他场景属性重建和解释底层场景。这种系统在计算机视觉、计算机图形和机器人领域的应用中至关重要，包括在自动驾驶汽车中。为了表示从输入图像观察到的3D世界，这种系统使用数学模型，近年来，由于神经网络的表现力和捕捉精细细节的能力，作为这种系统中使用的模型，神经网络非常受欢迎。然而，目前基于神经网络的场景表示只擅长对观察场景的特定条件进行建模，不能推广到新的场景，限制了它们在许多应用中的使用。例如，如果一辆自动驾驶汽车被训练成只使用晴天的图像来模拟场景，那么在雨天或雪天，汽车的感知系统可能会崩溃。该项目旨在引入新的场景建模技术，使机器能够以更通用的方式感知和重建3D场景。调查人员将把这项研究的结果整合到课程开发和学生建议中，并与教育和非营利组织合作，向代表性不足的学生教授人工智能、视觉和图形学。在这个项目中，研究人员将探索新的方法，使表示法能够编码更多的结构(例如光场)，并将它们植根于物理学。设计这样的表示需要人工智能、计算机视觉和计算机图形学的知识。主要创新包括一类新的场景表示，旨在将隐式神经表示捕获场景细节的能力与物理表示对场景结构进行建模的能力连接起来；使用新的参数化从原始图像和视频推断表示的新方法，以实现数据高效的自我监督学习；以及利用表示进行下游计算机视觉和图形任务(如交互设计和场景合成)的新方法。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Scene understanding systems take visual inputs, like images or videos, and reconstruct and interpret the underlying scene in terms of 3D structure, objects like cars and people, and other scene properties. Such systems are crucial in applications in computer vision, computer graphics, and robotics, including in self-driving cars. To represent the 3D world as observed from the input imagery, such systems use mathematical models, and in recent years neural networks have been very popular as the models used in such systems, due to their expressiveness and ability to capture fine details. However, current neural network-based scene representations are only good at modeling the specific conditions under which a scene was observed, and cannot generalize to new scenarios, limiting their use in many applications. For example, if a self-driving car is trained to model scenes using only images from sunny days, the car’s perception system might break down on rainy or snowy days. This project aims to introduce new scene modeling techniques that will enable machines to perceive and reconstruct 3D scenes in a more generalizable way. The investigators will integrate findings from this research into course development and student advising, and partner with educational and non-profit organizations to teach AI, vision, and graphics to underrepresented students. In this project, investigators will explore new methods that will make representations capable of encoding more structure (e.g., light field) and root them in physics. Designing such representations requires knowledge from AI, computer vision, and computer graphics. The key innovations include a new class of scene representations that aims to bridge the ability of implicit neural representations to capture scene details with that of physical representations to model scene structure; new methods that infer the representation from raw images and videos with new parametrizations to enable data-efficient, self-supervised learning; and new methods that leverage the representation for downstream computer vision and graphics tasks, such as interactive design and scene synthesis.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.