CAREER: Physically-Motivated Learning of 3D Shape and Semantics

导航或与真实的世界交互的系统必须推理3D几何属性，例如对象的距离或方向，以及语义属性，例如部件位置或对象类型。该项目将基于物理的图像和形状建模与鲁棒优化和深度学习的多功能性相结合，以恢复3D形状和语义信息。这项工作建立了计算机视觉、机器学习、计算机图形学和感知之间的联系。视觉是一种强大的感知方式，因为图像编码了关于形状和语义的丰富信息。然而，图像形成是一种物理现象，通常包括复杂的因素，如形状变形，遮挡，材料特性和参与介质。因此，自主或智能视觉系统的实际部署需要对各种物理因素的影响具有鲁棒性。这种效果可以通过对图像形成过程进行建模来逆转，但是手工制作的特征和硬编码的规则面临与模型不一致的数据的限制。深度学习的最新进展带来了令人印象深刻的性能，但推广纯数据驱动的方法来处理这种复杂的影响是昂贵的。为了应对这些挑战，该项目通过将图像形成的物理模型纳入深度学习框架来开发处理真实世界图像多样性的技术。该项目通过课程创建了视觉，图形，学习和感知方面的跨学科教育计划，该课程将基于物理的建模，深度学习，3D重建和语义理解等广泛领域联系起来。该项目还开发了K-12教育模块，为虚拟现实或自动驾驶等新技术提供体验式见解，重点关注来自代表性不足背景的学生。这项研究为3D形状和语义的物理动机学习奠定了基础，具有更高的准确性，更好的泛化或更易于培训等优点。它开发了将未知材料行为与3D形状相关联的理论框架，这允许稳健的优化框架和卷积神经网络架构用于材料不变的形状估计。它设计了新颖的网络结构，对复杂的变换进行建模，以概括非刚性和铰接变形或由于折射和参与介质引起的扭曲的形状或语义的恢复。此外，它使用外观或运动的物理模型来弥合模拟和真实的图像之间的域差距，从而产生减轻数据注释费用的弱监督框架。这些进步使光场成像、增强现实、恶劣天气下的自动驾驶或水下机器人探索等新应用成为可能。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

A system that navigates or interacts with the real world must reason about 3D geometric properties such as distances or orientations of objects, as well as semantic properties such as part locations or object types. This project combines physics-based modeling of images and shapes, with the versatility of robust optimization and deep learning, to recover 3D shape and semantic information. The work establishes connections between computer vision, machine learning, computer graphics and perception. Vision is a powerful sensing modality since images encode rich information about shape and semantics. However, image formation is a physical phenomenon that often includes complex factors like shape deformations, occlusions, material properties and participating media. Consequently, practical deployment of autonomous or intelligent vision-based systems requires robustness to the effects of diverse physical factors. Such effects may be inverted by modeling the image formation process, but hand-crafted features and hard-coded rules face limitations for data inconsistent with the model. Recent advances in deep learning have led to impressive performances, but generalization of a purely data-driven approach to handle such complex effects is expensive. To address these challenges, this project develops technologies of handling the diversity of real-world images through incorporation of physical models of image formation within deep learning frameworks. The project creates a cross-disciplinary educational program in vision, graphics, learning and perception through coursework that draws connections across wide areas such as physically-based modeling, deep learning, 3D reconstruction and semantic understanding. The program also develops K-12 educative modules that provide experiential insight into novel technologies such as virtual reality or self-driving, with a focus on outreach to students from under-represented backgrounds.This research lays the foundations for physically-motivated learning of 3D shape and semantics, with benefits such as higher accuracy, better generalization or greater ease of training. It develops theoretical frameworks that relate unknown material behavior to 3D shape, which allows robust optimization frameworks and convolutional neural network architectures for material-invariant shape estimation. It designs novel network structures that model complex transformations, to generalize recovery of shape or semantics across non-rigid and articulated deformations, or distortions due to refraction and participating media. Further, it uses physical models of appearance or motion to bridge the domain gap between simulations and real images, leading to weakly supervised frameworks that mitigate the expense of data annotation. These advances enable novel applications for light field imaging, augmented reality, self-driving in challenging weather, or underwater robotic exploration.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.