CRII: RI: Reasoning Geometric Commonsense for 3D Image/Video Parsing

常识推理研究绝大多数人可用的共识现实，知识，因果关系和基本原理，并可用于增强人工智能（AI）的各个方面。该项目开发几何常识的表示以及计算机视觉应用的常识推理的计算原理。该项目系统地研究了场景实体几何尺寸的常识知识，例如，轿车的长度比公共汽车的长度短;或者同一立面上的窗边相互平行并与地面上的边正交。这些一阶和二阶知识一旦被提取，在不同类型的场景中是相当稳定的，并且足以增强对2D和3D图像或视频的理解。该项目通过支持研究生，将研究与教育结合起来，并通过在相关会议上组织研讨会，向计算机视觉和人工智能研究社区伸出援手。本研究研究旨在研究图像或视频中3D场景解析的几何常识推理，并提供一种能够重建各种场景类别（例如，郊区、城市、校园）。 该项目从两个方面着手解决这一问题。首先，一个新的属性语法模型的开发表示图像和相关的几何常识知识，使用层次图形结构。利用该语法模型，通过生成有效的图像或视频解析图，可以实现语义区域的分割、场景实体的重构和几何常识的推理。其次，引入了一种新的计算框架，使得图像解析的推理可以在离散语义标签和连续几何标签的联合空间中进行，并且语法模型的学习可以在弱监督的训练图像上进行。所开发的技术使一个国家的最先进的计算机视觉系统，可以鲁棒地估计从图像或视频的语义和几何场景结构。

Commonsense reasoning studies the consensus reality, knowledge, causality, and rationales available to the overwhelming majority of people, and can be used to enhance all aspects of Artificial Intelligence (AI). This project develops representations of geometric commonsense as well as computing principles of commonsense reasoning for computer vision applications. The project systemically studies commonsense knowledge over geometric dimensions of scene entities, e.g., the length of a sedan is shorter than that of a bus; or that window edges on the same façade are parallel to each other and are orthogonal to the edges on the ground. These first-order and second-order knowledges, once extracted, are fairly stable across different types of scenes, and are informative enough for enhancing the understanding of images or videos in both 2D and 3D. The project integrates research with education by supporting graduate students, and outreaches to computer vision and AI research communities by organizing workshops in the relevant conferences.This research studies geometric commonsense reasoning for 3D scene parsing in images or videos, and contributes a unified probabilistic approach that is capable of reconstructing a wide variety of scene categories (e.g., suburb, urban, campus) from a single input image or a monocular video sequence. The project approaches the problem from two aspects. First, a new attributed grammar model is developed to represent both images and the associated geometric commonsense knowledge using a hierarchical graphical structure. With this grammar model, the segmentation of semantic regions, the reconstruction of scene entities, and the reasoning of geometric commonsense can be all solved through creating a valid parse graph from images or videos. Second, a new computing framework is introduced so that the inference of image parsing can be conducted in the joint space of discrete semantic labels and continuous geometric labels, and the learning of grammar models can be conducted over training images with weak supervision. The developed techniques enable a state-of-the-art computer vision system that can robustly estimate semantic and geometric scene structures from images or videos.