CAREER: Toward Discovering the 3D Geometrical and Semantic Structure of Objects and Scenes

该项目开发了一种新的框架，用于从图像中共同理解复杂场景的3D空间和语义结构。最先进的计算机视觉方法分别处理这两个任务。用于对象识别的方法通常将场景描述为类别标签的列表，但无法考虑3D空间结构。场景3D建模的方法产生精确的度量重建，但无法推断其组件的语义内容。该项目旨在填补这一空白，并为连贯地描述对象，对象组件及其在场景物理空间中的3D空间布置的新框架奠定基础。 该项目的研究主要有两个贡献。首先，新的模型表示的内在多视图性质的对象类别和测量关键对象的几何属性进行了探讨。其次，一个新的连贯的概率公式，能够使用这些测量同时估计最有可能的3D配置的场景元素和场景的关键语义现象进行了研究。这项研究有可能在许多战略领域发挥变革性作用，如自主导航，机器人和城市环境的3D自动建模。此外，它在设计帮助功能能力下降的人的技术方面至关重要。 该项目将研究和教育结合起来，让本科生或高中生参与主要应用目标是为残疾人开发技术的项目。

This project develops a novel framework for jointly understanding the 3D spatial and semantic structure of complex scenes from images. The state-of-the-art computer vision methods deal with these two tasks separately. Methods for object recognition typically describe the scene as a list of class labels, but are unable to account for the 3D spatial structure. Methods for scene 3D modeling produce accurate metric reconstructions but are unable to infer the semantic content of its components. This project seeks to fill this gap and creates the foundations for a new framework for coherently describing objects, object components and their 3D spatial arrangement in the scene's physical space. The research of this project makes two main contributions. First, novel models for representing the intrinsic multi-view nature of object categories and for measuring critical object geometrical attributes are explored. Second, a new coherent probabilistic formulation that is capable to use these measurements for simultaneously estimating the most likely 3D configuration of scene elements and the critical semantic phenomena of the scene are investigated. This research has potential to play a transformative role in many strategic areas such as autonomous navigation, robotics, and 3D automatic modeling of urban environments. Moreover, it is crucial in designing technology for assisting people with reduced functional capabilities. The project integrates research and education by involving undergraduates or high school students in projects whose primary application goal is to develop technology for people with disabilities.