RI: Small: Panoptic 3D Parsing in the Wild

由于使用了对2D投影的内在3D世界进行编码的有效表示(尚未完全理解)，人类具有识别/理解3D对象和场景的显著能力。计算机视觉的主要目标之一是开发能够“看到”世界的系统。这个项目指向一个新的方向，全景3D解析(Panoptic3D)，它联合对自然场景的单视RGB图像进行语义分割、目标检测、深度估计、3D形状重建和3D布局估计。2D和3D图像建模、表示学习、深层模型以及大规模跨通道数据集的快速发展为构建Panoptic3D系统提供了前所未有的机遇。Panoptic3D系统可用于为计算机科学以外的其他学科的科学研究和实验提供帮助，如认知科学、神经科学、医疗保健、交通/土木工程、机械工程和计算生物学。该项目重点介绍了构建一种新系统-全景3D解析(Panoptic3D)的路线图，该系统联合对野外单视RGB图像执行语义分割、对象检测、实例分割、深度估计、3D形状重建和3D布局估计。单视图像的图像理解和三维(形状与布局)重建问题是计算机视觉和摄影测量学几十年发展中根深蒂固的问题。该项目的灵感来自于在整体图像理解和单视图三维形状/布局重建方面的最新发展，大规模2D/3D图像数据集的可获得性，以及深度学习和表征学习的成功。将通过开发新的3D建模和计算算法来进行一些技术创新，以解决自然图像的分割/对象/3D形状/3D布局缺乏全面的多形态地面真实注释集的问题。追求这一新方向的潜在收益是巨大的，拟议的Panoptic3D系统适用于一系列领域，包括计算机视觉、计算机图形、自动驾驶、地图绘制、机器人、人机交互和增强现实。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Humans have the remarkable capability of recognizing/understanding 3D objects and scenes, due to the use of effective representations (yet not fully understood) that encode the intrinsic 3D world for the 2D projections. One of the main objectives in computer vision is to develop systems that can "see" the world. This project points to a new direction, panoptic 3D parsing (Panoptic3D), that jointly performs semantic segmentation, object detection, depth estimation, 3D shape reconstruction, and 3D layout estimation for single-view RGB images of natural scenes. The rapid development in 2D and 3D image modeling, representation learning, deep models, as well as large-scale cross-modality datasets provides an unprecedented opportunity to building the Panoptic3D systems. The Panoptic3D system can be adopted to offer assistance to scientific studies and experiments in other disciplines beyond computer science such as cognitive science, neuroscience, health-care, transportation/civil engineering, mechanical engineering, and computational biology. This project highlights a roadmap to building a novel system, Panoptic 3D Parsing (Panoptic3D), that jointly performs semantic segmentation, object detection, instance segmentation, depth estimation, 3D shape reconstruction, and 3D layout estimation for single-view RGB images in the wild. The problem of image understanding and 3D (shapes and layout) reconstruction for single-view image is deeply rooted in decades of development in computer vision and photogrammetry. The project is inspired by the recent development in holistic image understanding and single-view 3D shape/layout reconstruction, the availability of large-scale 2D/3D image datasets, as well as successes in deep learning and representation learning. A number of technical innovations will be made by developing new 3D modeling and computing algorithms when combating the issue of absent comprehensive sets of multi-modality ground-truth annotations for segmentation/objects/3D shapes/3D layout of natural images in the wild. The potential gain of pursing this new direction is substantial and the proposed Panoptic3D system is applicable to a range of domains including computer vision, computer graphics, autonomous driving, mapping, robotics, human-computer interaction, and augmented reality.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.