RI: Small: 3D Reconstruction via Differential Rendering and Deep Learning

数字重建现实世界物体的3D形状是一项核心技术，可以实现非常广泛的应用，例如自主机器人导航；用于个人用途或逆向工程的3D打印；存档和虚拟遗产；为电影、游戏、增强现实和虚拟现实创造资产；或大规模重建的地理信息系统。该项目开发了一种新的计算机算法，利用数字图像作为输入来重建物体的三维形状。它解决了当前技术在实际应用中经常导致不准确结果的重大局限性。为了实现这一目标，该项目采用了一种创新的方法，利用人工智能技术来理解基于数字图像的3D形状。利用人工智能方法进行三维形状重建是一项重要的科学进步，有望在研究领域取得进一步进展。由学生领导的增强现实（AR）和虚拟现实（VR）俱乐部获得了基于人工智能的3D重建的最新研究和实验的第一手经验，以设计创新的AR和VR应用程序。本研究开发了基于两个关键技术的算法，可微渲染和深度学习。结合这两种方法可以产生协同效应，从而克服当前算法的局限性。渲染是通过算法评估图像形成模型的过程，其中可能包括复杂的光传输效果，如非漫射表面、阴影和间接照明，以计算虚拟3D对象或环境的图像。利用自动微分（AD），可微分渲染器计算渲染图像的像素值相对于虚拟三维模型的所有未知模型参数的偏导数。利用AD和可微分渲染的功能和通用性，可以克服以前工作中常见的过于简单的图像形成模型。此外，由于一组视图中存在大量的未知参数和有限的信息，多视图重构往往是病态的。因此，需要强先验和鲁棒误差度量。这项工作使用大规模形状和图像数据库以及深度学习技术获得这些误差度量和先验，以捕获现实世界物体的全部复杂性。至关重要的是，它通过可微分渲染将深度学习连接到未知的3D模型参数，这使得利用基于梯度的优化技术来求解所需的3D形状成为可能。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Digitally reconstructing the 3D shapes of real-world objects is a core technology that enables a very wide range of applications, such as autonomous robot navigation; 3D printing for personal purposes or reverse engineering; archiving and virtual heritage; creating assets for movies, games, and augmented and virtual reality; or large-scale reconstruction for geographical information systems. This project develops novel computer algorithms to reconstruct the 3D shapes of objects using digital images as inputs. It addresses significant limitations of current techniques that often lead to inaccurate results in real-life applications. To achieve this, the project follows an innovative approach leveraging artificial intelligence techniques to understand 3D shapes based on digital images. The formulation of 3D shape reconstruction using artificial intelligence methods represents an important scientific advancement that promises further advances in the research field. A student-led augmented reality (AR) and virtual reality (VR) club gains first-hand experience with state of the art research and experiments with artificial intelligence-based 3D reconstruction to design innovative AR and VR applications.This research develops algorithms building on two key techniques, differentiable rendering and deep learning. Combining these two methods leads to synergies that can overcome the limitations of current algorithms. Rendering is the process of algorithmically evaluating an image formation model, which may include sophisticated light transport effects such as non-diffuse surfaces, shadows, and indirect illumination, to compute an image of a virtual 3D object or environment. Using automatic differentiation (AD), a differentiable renderer calculates the partial derivatives of pixel values of rendered images with respect to all unknown model parameters of the virtual 3D model. Leveraging the power and generality of AD and differentiable rendering allows to overcome the overly simplistic image formation models common in previous work. In addition, multi-view reconstruction is often ill-posed because of the large number of unknown parameters and the limited information present in a set of views. Therefore, strong priors and robust error metrics are required. This work obtains these error metrics and priors using large-scale shape and image databases and deep learning techniques, to capture the full complexity of real-world objects. Crucially, it connects deep learning to the unknown 3D model parameters through differentiable rendering, which makes it possible to leverage gradient-based optimization techniques to solve for the desired 3D shapes.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.