CAREER: Physically-Motivated Learning of 3D Shape and Semantics

导航或与现实世界交互的系统必须推理3D几何属性，如物体的距离或方向，以及语义属性，如零件位置或对象类型。该项目结合了基于物理的图像和形状建模，以及鲁棒优化和深度学习的多功能性，以恢复3D形状和语义信息。这项工作建立了计算机视觉、机器学习、计算机图形学和感知之间的联系。视觉是一种强大的感知方式，因为图像编码了丰富的形状和语义信息。然而，图像的形成是一种物理现象，通常包括形状变形、遮挡、材料特性和参与介质等复杂因素。因此，自主或智能视觉系统的实际部署需要对各种物理因素的影响具有鲁棒性。这种效果可以通过对图像形成过程进行建模来逆转，但是手工制作的特征和硬编码的规则面临与模型不一致的数据的限制。深度学习的最新进展带来了令人印象深刻的表现，但将纯数据驱动的方法推广到处理如此复杂的效果是昂贵的。为了应对这些挑战，该项目通过在深度学习框架中结合图像形成的物理模型，开发了处理现实世界图像多样性的技术。该项目在视觉、图形、学习和感知方面创建了一个跨学科的教育项目，通过课程作业，将基于物理的建模、深度学习、3D重建和语义理解等广泛领域联系起来。该项目还开发了K-12教育模块，提供对虚拟现实或自动驾驶等新技术的体验性见解，重点是向来自代表性不足背景的学生提供服务。这项研究为三维形状和语义的物理动机学习奠定了基础，具有更高的准确性，更好的泛化或更容易训练等优点。它开发了将未知材料行为与3D形状联系起来的理论框架，这允许用于材料不变形状估计的鲁棒优化框架和卷积神经网络架构。它设计了新颖的网络结构来模拟复杂的转换，以推广非刚性和铰接变形的形状或语义恢复，或由于折射和参与媒体造成的扭曲。此外，它使用外观或运动的物理模型来弥合模拟和真实图像之间的领域差距，从而产生弱监督框架，从而减轻数据注释的费用。这些进步使光场成像、增强现实、恶劣天气下的自动驾驶或水下机器人探索等新应用成为可能。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

A system that navigates or interacts with the real world must reason about 3D geometric properties such as distances or orientations of objects, as well as semantic properties such as part locations or object types. This project combines physics-based modeling of images and shapes, with the versatility of robust optimization and deep learning, to recover 3D shape and semantic information. The work establishes connections between computer vision, machine learning, computer graphics and perception. Vision is a powerful sensing modality since images encode rich information about shape and semantics. However, image formation is a physical phenomenon that often includes complex factors like shape deformations, occlusions, material properties and participating media. Consequently, practical deployment of autonomous or intelligent vision-based systems requires robustness to the effects of diverse physical factors. Such effects may be inverted by modeling the image formation process, but hand-crafted features and hard-coded rules face limitations for data inconsistent with the model. Recent advances in deep learning have led to impressive performances, but generalization of a purely data-driven approach to handle such complex effects is expensive. To address these challenges, this project develops technologies of handling the diversity of real-world images through incorporation of physical models of image formation within deep learning frameworks. The project creates a cross-disciplinary educational program in vision, graphics, learning and perception through coursework that draws connections across wide areas such as physically-based modeling, deep learning, 3D reconstruction and semantic understanding. The program also develops K-12 educative modules that provide experiential insight into novel technologies such as virtual reality or self-driving, with a focus on outreach to students from under-represented backgrounds.This research lays the foundations for physically-motivated learning of 3D shape and semantics, with benefits such as higher accuracy, better generalization or greater ease of training. It develops theoretical frameworks that relate unknown material behavior to 3D shape, which allows robust optimization frameworks and convolutional neural network architectures for material-invariant shape estimation. It designs novel network structures that model complex transformations, to generalize recovery of shape or semantics across non-rigid and articulated deformations, or distortions due to refraction and participating media. Further, it uses physical models of appearance or motion to bridge the domain gap between simulations and real images, leading to weakly supervised frameworks that mitigate the expense of data annotation. These advances enable novel applications for light field imaging, augmented reality, self-driving in challenging weather, or underwater robotic exploration.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.