RI: Small: 3D Reconstruction via Differential Rendering and Deep Learning

数字重建现实世界物体的3D形状是一项核心技术，可以实现非常广泛的应用，例如自主机器人导航;用于个人目的或逆向工程的3D打印;存档和虚拟遗产;为电影，游戏以及增强和虚拟现实创建资产;或地理信息系统的大规模重建。该项目开发新的计算机算法，以数字图像作为输入来重建物体的3D形状。它解决了当前技术的重大局限性，这些局限性往往导致在现实生活中的应用程序中产生不准确的结果。为了实现这一目标，该项目采用了一种创新的方法，利用人工智能技术来理解基于数字图像的3D形状。使用人工智能方法的3D形状重建的制定代表了一个重要的科学进步，有望在研究领域取得进一步的进展。一个由学生主导的增强现实（AR）和虚拟现实（VR）俱乐部通过最先进的研究和基于人工智能的3D重建实验获得第一手经验，以设计创新的AR和VR应用程序。这项研究开发了基于两个关键技术的算法，可区分渲染和深度学习。将这两种方法结合起来，可以克服当前算法的局限性。渲染是在算法上评估图像形成模型的过程，该图像形成模型可以包括复杂的光传输效果，诸如非漫射表面、阴影和间接照明，以计算虚拟3D对象或环境的图像。使用自动微分（AD），可微分渲染器计算渲染图像的像素值相对于虚拟3D模型的所有未知模型参数的偏导数。利用AD和微分渲染的能力和通用性，可以克服以前工作中常见的过于简单的图像形成模型。此外，多视图重建往往是不适定的，因为大量的未知参数和有限的信息存在于一组视图。因此，需要强先验和鲁棒的误差度量。这项工作使用大规模形状和图像数据库以及深度学习技术来获取这些误差度量和先验，以捕获真实世界对象的全部复杂性。重要的是，它通过可微分渲染将深度学习与未知的3D模型参数联系起来，从而可以利用基于梯度的优化技术来求解所需的3D形状。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Digitally reconstructing the 3D shapes of real-world objects is a core technology that enables a very wide range of applications, such as autonomous robot navigation; 3D printing for personal purposes or reverse engineering; archiving and virtual heritage; creating assets for movies, games, and augmented and virtual reality; or large-scale reconstruction for geographical information systems. This project develops novel computer algorithms to reconstruct the 3D shapes of objects using digital images as inputs. It addresses significant limitations of current techniques that often lead to inaccurate results in real-life applications. To achieve this, the project follows an innovative approach leveraging artificial intelligence techniques to understand 3D shapes based on digital images. The formulation of 3D shape reconstruction using artificial intelligence methods represents an important scientific advancement that promises further advances in the research field. A student-led augmented reality (AR) and virtual reality (VR) club gains first-hand experience with state of the art research and experiments with artificial intelligence-based 3D reconstruction to design innovative AR and VR applications.This research develops algorithms building on two key techniques, differentiable rendering and deep learning. Combining these two methods leads to synergies that can overcome the limitations of current algorithms. Rendering is the process of algorithmically evaluating an image formation model, which may include sophisticated light transport effects such as non-diffuse surfaces, shadows, and indirect illumination, to compute an image of a virtual 3D object or environment. Using automatic differentiation (AD), a differentiable renderer calculates the partial derivatives of pixel values of rendered images with respect to all unknown model parameters of the virtual 3D model. Leveraging the power and generality of AD and differentiable rendering allows to overcome the overly simplistic image formation models common in previous work. In addition, multi-view reconstruction is often ill-posed because of the large number of unknown parameters and the limited information present in a set of views. Therefore, strong priors and robust error metrics are required. This work obtains these error metrics and priors using large-scale shape and image databases and deep learning techniques, to capture the full complexity of real-world objects. Crucially, it connects deep learning to the unknown 3D model parameters through differentiable rendering, which makes it possible to leverage gradient-based optimization techniques to solve for the desired 3D shapes.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.