CAREER: Toward Discovering the 3D Geometrical and Semantic Structure of Objects and Scenes

该项目开发了一个新的框架，用于从图像中共同理解复杂场景的三维空间和语义结构。最先进的计算机视觉方法分别处理这两项任务。物体识别方法通常将场景描述为类标签列表，但无法解释3D空间结构。场景三维建模方法产生精确的度量重建，但无法推断其组成部分的语义内容。该项目旨在填补这一空白，并为连贯地描述物体、物体组件及其在场景物理空间中的3D空间安排的新框架奠定基础。本项目的研究有两个主要贡献。首先，探索了表征对象类别固有的多视图性质和测量关键对象几何属性的新模型。其次，研究了一种新的连贯概率公式，该公式能够使用这些测量值同时估计场景元素最可能的3D配置和场景的关键语义现象。这项研究有可能在许多战略领域发挥变革性作用，如自主导航、机器人技术和城市环境的3D自动建模。此外，它在设计技术以帮助有功能障碍的人是至关重要的。该项目将研究和教育结合起来，让本科生或高中生参与项目，其主要应用目标是为残疾人开发技术。

This project develops a novel framework for jointly understanding the 3D spatial and semantic structure of complex scenes from images. The state-of-the-art computer vision methods deal with these two tasks separately. Methods for object recognition typically describe the scene as a list of class labels, but are unable to account for the 3D spatial structure. Methods for scene 3D modeling produce accurate metric reconstructions but are unable to infer the semantic content of its components. This project seeks to fill this gap and creates the foundations for a new framework for coherently describing objects, object components and their 3D spatial arrangement in the scene's physical space. The research of this project makes two main contributions. First, novel models for representing the intrinsic multi-view nature of object categories and for measuring critical object geometrical attributes are explored. Second, a new coherent probabilistic formulation that is capable to use these measurements for simultaneously estimating the most likely 3D configuration of scene elements and the critical semantic phenomena of the scene are investigated. This research has potential to play a transformative role in many strategic areas such as autonomous navigation, robotics, and 3D automatic modeling of urban environments. Moreover, it is crucial in designing technology for assisting people with reduced functional capabilities. The project integrates research and education by involving undergraduates or high school students in projects whose primary application goal is to develop technology for people with disabilities.