RI: Small: 3D Reconstruction via Differential Rendering and Deep Learning

数字重建真实世界对象的3D形状是一项核心技术，它实现了非常广泛的应用，如自主机器人导航；用于个人目的或逆向工程的3D打印；存档和虚拟遗产；为电影、游戏和增强现实创建资源；或地理信息系统的大规模重建。该项目开发了新的计算机算法，使用数字图像作为输入来重建物体的3D形状。它解决了当前技术的重大局限性，这些局限性经常导致实际应用中的不准确结果。为了实现这一目标，该项目采用了一种创新的方法，利用人工智能技术来理解基于数字图像的3D形状。使用人工智能方法进行三维形状重建的提出代表了一项重要的科学进步，有望在该研究领域取得进一步进展。一个由学生领导的增强现实(AR)和虚拟现实(VR)俱乐部获得了基于人工智能的3D重建的第一手经验和最先进的研究，以设计创新的AR和VR应用程序。这项研究开发了基于两个关键技术的算法，可微渲染和深度学习。将这两种方法结合起来可以产生协同效应，可以克服当前算法的局限性。渲染是对图像形成模型进行算法评估的过程，该模型可能包括复杂的光传输效果，例如非漫反射表面、阴影和间接照明，以计算虚拟3D对象或环境的图像。使用自动微分(AD)，可微绘制器计算渲染图像的像素值相对于虚拟3D模型的所有未知模型参数的偏导数。利用AD和可微渲染的能力和通用性，可以克服以前工作中常见的过于简单的图像形成模型。此外，由于一组视图中存在大量的未知参数和有限的信息，因此多视图重建往往是不适定的。因此，需要强大的先验和稳健的误差度量。这项工作使用大规模的形状和图像数据库以及深度学习技术来获取这些误差度量和先验知识，以捕捉真实世界对象的全部复杂性。最重要的是，它通过可微渲染将深度学习与未知的3D模型参数联系起来，这使得利用基于梯度的优化技术来求解所需的3D形状成为可能。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Digitally reconstructing the 3D shapes of real-world objects is a core technology that enables a very wide range of applications, such as autonomous robot navigation; 3D printing for personal purposes or reverse engineering; archiving and virtual heritage; creating assets for movies, games, and augmented and virtual reality; or large-scale reconstruction for geographical information systems. This project develops novel computer algorithms to reconstruct the 3D shapes of objects using digital images as inputs. It addresses significant limitations of current techniques that often lead to inaccurate results in real-life applications. To achieve this, the project follows an innovative approach leveraging artificial intelligence techniques to understand 3D shapes based on digital images. The formulation of 3D shape reconstruction using artificial intelligence methods represents an important scientific advancement that promises further advances in the research field. A student-led augmented reality (AR) and virtual reality (VR) club gains first-hand experience with state of the art research and experiments with artificial intelligence-based 3D reconstruction to design innovative AR and VR applications.This research develops algorithms building on two key techniques, differentiable rendering and deep learning. Combining these two methods leads to synergies that can overcome the limitations of current algorithms. Rendering is the process of algorithmically evaluating an image formation model, which may include sophisticated light transport effects such as non-diffuse surfaces, shadows, and indirect illumination, to compute an image of a virtual 3D object or environment. Using automatic differentiation (AD), a differentiable renderer calculates the partial derivatives of pixel values of rendered images with respect to all unknown model parameters of the virtual 3D model. Leveraging the power and generality of AD and differentiable rendering allows to overcome the overly simplistic image formation models common in previous work. In addition, multi-view reconstruction is often ill-posed because of the large number of unknown parameters and the limited information present in a set of views. Therefore, strong priors and robust error metrics are required. This work obtains these error metrics and priors using large-scale shape and image databases and deep learning techniques, to capture the full complexity of real-world objects. Crucially, it connects deep learning to the unknown 3D model parameters through differentiable rendering, which makes it possible to leverage gradient-based optimization techniques to solve for the desired 3D shapes.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.