Collaborative Research: HCC: Medium: Differentiable Rendering for Computer Graphics

在计算机图形学中创建逼真的图像在历史上依赖于基于场景中的照明的物理上精确的模拟来精确地计算图像中的每个点或像素处的值，但是最近已经变得清楚的是，简单地计算图像值是不够的。人们还需要能够理解这些值如何随着环境的变化而变化，例如，当太阳在天空中移动时，打开一扇门让光线进入场景，或者物体的材料属性逐渐从天鹅绒变为金属。 在数学上，这涉及计算图像的导数以确定其相对于输入参数如何变化。 这项研究将创建一类可微分渲染器，计算图像及其衍生物。 项目成果将产生广泛的影响，因为导数的计算在计算机图形学、计算机视觉、机器人和机器学习的许多领域越来越重要，对各种应用都有潜在的好处，如自动驾驶汽车和机器人的感知控制、建筑室内照明的优化、具有理想外观的3D物体的制造、统计学和流行病学。 其他影响将来自于这样一个事实，即PI是教育工作者，致力于扩大参与计算谁参加早期的研究学者计划，并将开发新的在线课程在rendering.Computing一般光传输的导数或梯度涉及解决微分学，蒙特卡罗积分，信号处理，自动微分和元编程系统的基本挑战。 一个挑战是在处理各种形式的不连续性，这导致狄拉克δ项，需要仔细和分析的处理，不能提供传统的自动微分。 即使对于平滑变化，计算梯度也涉及大量中间变量，这些中间变量需要在偏差、方差、计算和存储器之间进行权衡。 此外，充分的一般性需要微分渲染在新的表示，如隐式表面和程序材料，以及新的问题域，如瞬态渲染的非视线成像和几何衍射声学。 人们还需要有效地应用梯度来优化逆问题。 该项目将制定一个广泛的变革议程，寻求使可区分的渲染器能够在一般和不同的光传输情况下从数百万像素有效地重建数十亿个不同的图元。 该研究计划包括四个相互关联的组成部分，涉及计算基础和解决可见度梯度的有效算法，包括：分析和区域采样方法;探索可微渲染算法中计算和内存权衡的统一系统;对新物理现象的概括，如瞬态渲染和几何衍射;以及逆问题和深度学习的进展，包括涉及欧拉的连续优化的新方法，该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的评估来支持。影响审查标准。

Creating realistic images in computer graphics has historically relied on accurately computing the values at each point or pixel in the image based on physically accurate simulation of lighting in the scene, but recently it has become clear that simply computing image values is not adequate. One needs to also be able to understand how these values change with changes in the environment, for example as the sun moves across the sky, a door is opened letting light into the scene, or the material properties of an object are gradually changed from velvet to metallic. Mathematically, this involves computing the derivatives of the image to determine how it changes with respect to the input parameters. This research will create a class of differentiable renderers that compute both images and their derivatives. Project outcomes will have broad impact because the computation of derivatives is increasingly central to many areas of computer graphics, computer vision, robotics and machine learning, with potential benefit to applications as diverse as perception control in self-driving cars and robots, optimization of indoor lighting for architecture, fabrication of 3D objects with a desired appearance, statistics and epidemiology. Additional impact will derive from the fact that the PIs are educators committed to broadening participation in computing who participate in early research scholars programs and will develop new online courses in rendering.Computing the derivatives or gradients of general light transport involves tackling fundamental challenges of differential calculus, Monte Carlo integration, signal processing, automatic differentiation, and metaprogramming systems. One challenge is in handling discontinuities of various forms, which lead to Dirac delta terms that require careful and analytic treatment that cannot be provided by traditional automatic differentiation. Even for the smooth variation, computing gradients involves a large number of intermediate variables that necessitate tradeoffs across bias, variance, compute and memory. Moreover, full generality requires differentiable rendering in new representations such as implicit surfaces and procedural materials, as well as new problem domains such as transient rendering for non-line-of-sight imaging and geometrical diffraction for acoustics. One also needs to effectively apply the gradients for optimization in inverse problems. This project will develop a broad transformative agenda, seeking to enable differentiable renderers to efficiently reconstruct billions of varied primitives from millions of pixels under general and diverse light transport situations. The research plan consists of four interconnected components involving computational foundations and efficient algorithms for solving visibility gradients including: analytic and area sampling methods; a unified system for exploring computational and memory tradeoffs in differentiable rendering algorithms; generalizations to new physical phenomena such as transient rendering and geometrical diffraction; and advances in inverse problems and deep learning including new approaches to continuous optimization involving Euler-Lagrange equations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.