Collaborative Research: HCC: Medium: Differentiable Rendering for Computer Graphics

在计算机图形学中创建逼真的图像历来依赖于精确计算图像中每个点或像素的值，这是基于对场景中照明的物理精确模拟，但最近很明显，仅仅计算图像值是不够的。人们还需要能够理解这些值是如何随着环境的变化而变化的，例如，当太阳穿过天空时，一扇门被打开，让光线进入场景，或者物体的材料属性逐渐从天鹅绒变成金属。在数学上，这包括计算图像的导数，以确定它如何随输入参数变化。这项研究将创建一类可微分渲染器，用于计算图像及其衍生物。项目成果将产生广泛的影响，因为衍生品的计算在计算机图形学、计算机视觉、机器人技术和机器学习的许多领域越来越重要，对自动驾驶汽车和机器人的感知控制、建筑室内照明的优化、具有理想外观的3D物体的制造、统计和流行病学等各种应用都有潜在的好处。额外的影响将来自于这样一个事实，即pi是致力于扩大计算机参与的教育工作者，他们参与了早期的研究学者计划，并将开发新的渲染在线课程。计算一般光输运的导数或梯度涉及解决微分学、蒙特卡罗积分、信号处理、自动微分和元编程系统的基本挑战。一个挑战是处理各种形式的不连续，这导致狄拉克δ项需要仔细和解析处理，这是传统的自动微分所不能提供的。即使对于平滑的变化，计算梯度也涉及大量的中间变量，需要在偏差、方差、计算和内存之间进行权衡。此外，全面的一般性需要在新的表示中进行可微分渲染，例如隐式表面和程序材料，以及新的问题领域，例如非视距成像的瞬态渲染和声学的几何衍射。人们还需要有效地应用梯度来优化反问题。该项目将发展一个广泛的变革议程，寻求使可微分渲染器能够在一般和不同的光传输情况下，从数百万像素有效地重建数十亿种不同的原语。该研究计划由四个相互关联的部分组成，涉及求解能见度梯度的计算基础和有效算法，包括：解析和面积采样方法；在可微分渲染算法中探索计算和内存权衡的统一系统；对新的物理现象的概括，如瞬态渲染和几何衍射；在反问题和深度学习方面的进展，包括涉及欧拉-拉格朗日方程的连续优化的新方法。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Creating realistic images in computer graphics has historically relied on accurately computing the values at each point or pixel in the image based on physically accurate simulation of lighting in the scene, but recently it has become clear that simply computing image values is not adequate. One needs to also be able to understand how these values change with changes in the environment, for example as the sun moves across the sky, a door is opened letting light into the scene, or the material properties of an object are gradually changed from velvet to metallic. Mathematically, this involves computing the derivatives of the image to determine how it changes with respect to the input parameters. This research will create a class of differentiable renderers that compute both images and their derivatives. Project outcomes will have broad impact because the computation of derivatives is increasingly central to many areas of computer graphics, computer vision, robotics and machine learning, with potential benefit to applications as diverse as perception control in self-driving cars and robots, optimization of indoor lighting for architecture, fabrication of 3D objects with a desired appearance, statistics and epidemiology. Additional impact will derive from the fact that the PIs are educators committed to broadening participation in computing who participate in early research scholars programs and will develop new online courses in rendering.Computing the derivatives or gradients of general light transport involves tackling fundamental challenges of differential calculus, Monte Carlo integration, signal processing, automatic differentiation, and metaprogramming systems. One challenge is in handling discontinuities of various forms, which lead to Dirac delta terms that require careful and analytic treatment that cannot be provided by traditional automatic differentiation. Even for the smooth variation, computing gradients involves a large number of intermediate variables that necessitate tradeoffs across bias, variance, compute and memory. Moreover, full generality requires differentiable rendering in new representations such as implicit surfaces and procedural materials, as well as new problem domains such as transient rendering for non-line-of-sight imaging and geometrical diffraction for acoustics. One also needs to effectively apply the gradients for optimization in inverse problems. This project will develop a broad transformative agenda, seeking to enable differentiable renderers to efficiently reconstruct billions of varied primitives from millions of pixels under general and diverse light transport situations. The research plan consists of four interconnected components involving computational foundations and efficient algorithms for solving visibility gradients including: analytic and area sampling methods; a unified system for exploring computational and memory tradeoffs in differentiable rendering algorithms; generalizations to new physical phenomena such as transient rendering and geometrical diffraction; and advances in inverse problems and deep learning including new approaches to continuous optimization involving Euler-Lagrange equations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.