Collaborative Research: HCC: Medium: Differentiable Rendering for Computer Graphics

在计算机图形学中创建逼真的图像历来依赖于基于场景中照明的物理精确模拟来准确计算图像中每个点或像素处的值，但最近变得明显的是，简单地计算图像值是不够的。人们还需要能够理解这些值如何随着环境的变化而变化，例如，当太阳穿过天空时，打开一扇门让光线进入场景，或者对象的材质属性逐渐从天鹅绒变为金属。从数学上讲，这涉及计算图像的导数，以确定它如何相对于输入参数变化。这项研究将创建一类可以同时计算图像及其导数的可微渲染器。项目成果将产生广泛的影响，因为导数的计算在计算机图形学、计算机视觉、机器人学和机器学习的许多领域越来越重要，潜在的好处包括自动驾驶汽车和机器人的感知控制、建筑室内照明的优化、制造具有所需外观的3D对象、统计学和流行病学。其他影响将来自这样一个事实，即PI是致力于扩大对计算的参与的教育工作者，他们参与了早期研究学者计划，并将开发新的渲染在线课程。计算一般光传输的导数或梯度涉及解决微分学、蒙特卡罗积分、信号处理、自动微分和元编程系统的基本挑战。一个挑战是处理各种形式的不连续，这些不连续导致狄拉克三角洲项，这些项需要仔细和分析处理，而传统的自动微分无法提供。即使对于平滑的变化，计算梯度也涉及大量的中间变量，这需要在偏差、方差、计算和内存之间进行权衡。此外，完全的通用性要求在隐式曲面和程序材质等新表示中可微绘制，以及非视线成像的瞬变绘制和声学的几何绕射等新的问题领域。我们还需要在反问题中有效地应用优化的梯度。该项目将制定一个广泛的变革性议程，寻求使可区分的渲染器能够在一般和不同的光传输情况下从数百万像素高效地重建数十亿个不同的基本体。研究计划由四个相互关联的部分组成，涉及计算基础和求解能见度梯度的有效算法，包括：解析和面积采样方法；探索可微渲染算法中计算和内存权衡的统一系统；对新物理现象的概括，如瞬变渲染和几何绕射；反问题和深度学习方面的进展，包括涉及欧拉-拉格朗日方程的连续优化的新方法。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Creating realistic images in computer graphics has historically relied on accurately computing the values at each point or pixel in the image based on physically accurate simulation of lighting in the scene, but recently it has become clear that simply computing image values is not adequate. One needs to also be able to understand how these values change with changes in the environment, for example as the sun moves across the sky, a door is opened letting light into the scene, or the material properties of an object are gradually changed from velvet to metallic. Mathematically, this involves computing the derivatives of the image to determine how it changes with respect to the input parameters. This research will create a class of differentiable renderers that compute both images and their derivatives. Project outcomes will have broad impact because the computation of derivatives is increasingly central to many areas of computer graphics, computer vision, robotics and machine learning, with potential benefit to applications as diverse as perception control in self-driving cars and robots, optimization of indoor lighting for architecture, fabrication of 3D objects with a desired appearance, statistics and epidemiology. Additional impact will derive from the fact that the PIs are educators committed to broadening participation in computing who participate in early research scholars programs and will develop new online courses in rendering.Computing the derivatives or gradients of general light transport involves tackling fundamental challenges of differential calculus, Monte Carlo integration, signal processing, automatic differentiation, and metaprogramming systems. One challenge is in handling discontinuities of various forms, which lead to Dirac delta terms that require careful and analytic treatment that cannot be provided by traditional automatic differentiation. Even for the smooth variation, computing gradients involves a large number of intermediate variables that necessitate tradeoffs across bias, variance, compute and memory. Moreover, full generality requires differentiable rendering in new representations such as implicit surfaces and procedural materials, as well as new problem domains such as transient rendering for non-line-of-sight imaging and geometrical diffraction for acoustics. One also needs to effectively apply the gradients for optimization in inverse problems. This project will develop a broad transformative agenda, seeking to enable differentiable renderers to efficiently reconstruct billions of varied primitives from millions of pixels under general and diverse light transport situations. The research plan consists of four interconnected components involving computational foundations and efficient algorithms for solving visibility gradients including: analytic and area sampling methods; a unified system for exploring computational and memory tradeoffs in differentiable rendering algorithms; generalizations to new physical phenomena such as transient rendering and geometrical diffraction; and advances in inverse problems and deep learning including new approaches to continuous optimization involving Euler-Lagrange equations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.