CAREER: Toward Discovering the 3D Geometrical and Semantic Structure of Objects and Scenes

该项目开发了一种新的框架，用于从图像中联合理解复杂场景的3D空间和语义结构。最先进的计算机视觉方法分别处理这两个任务。用于对象识别的方法通常将场景描述为类别标签列表，但不能考虑3D空间结构。场景3D建模的方法产生准确的度量重建，但无法推断其组件的语义内容。该项目试图填补这一空白，并为在场景的物理空间中连贯地描述对象、对象组件及其3D空间排列的新框架奠定基础。本课题的研究主要有两个方面的贡献。首先，探索了用于表示对象类别固有的多视点性质和用于测量关键对象几何属性的新模型。其次，研究了一种新的相干概率公式，该公式能够使用这些测量来同时估计场景元素的最可能的3D构形和场景的关键语义现象。这项研究有可能在许多战略领域发挥变革性的作用，如自主导航、机器人和城市环境的3D自动建模。此外，它在设计帮助功能能力降低的人的技术方面也是至关重要的。该项目通过让本科生或高中生参与其主要应用目标是为残疾人开发技术的项目，将研究和教育结合在一起。

This project develops a novel framework for jointly understanding the 3D spatial and semantic structure of complex scenes from images. The state-of-the-art computer vision methods deal with these two tasks separately. Methods for object recognition typically describe the scene as a list of class labels, but are unable to account for the 3D spatial structure. Methods for scene 3D modeling produce accurate metric reconstructions but are unable to infer the semantic content of its components. This project seeks to fill this gap and creates the foundations for a new framework for coherently describing objects, object components and their 3D spatial arrangement in the scene's physical space. The research of this project makes two main contributions. First, novel models for representing the intrinsic multi-view nature of object categories and for measuring critical object geometrical attributes are explored. Second, a new coherent probabilistic formulation that is capable to use these measurements for simultaneously estimating the most likely 3D configuration of scene elements and the critical semantic phenomena of the scene are investigated. This research has potential to play a transformative role in many strategic areas such as autonomous navigation, robotics, and 3D automatic modeling of urban environments. Moreover, it is crucial in designing technology for assisting people with reduced functional capabilities. The project integrates research and education by involving undergraduates or high school students in projects whose primary application goal is to develop technology for people with disabilities.