CAREER: Physics-Based Differentiable and Inverse Rendering

该项目将开发新的计算工具，从测量图像（如照片）中推断物体形状和场景光学特性等物理参数。这些工具对于构建现实世界对象的数字双胞胎至关重要，并将在计算机视觉、计算成像、机器人和虚拟/增强现实（VR/AR）等广泛领域实现新的应用。与许多现有的纯粹由数据驱动的方法不同，这项研究将开发利用光如何传播的模拟器的推理技术。该模拟器将是可微的，这意味着它可以平滑地将其控制参数与其决策联系起来，为在诸如反射光照明等复杂场景下可靠地恢复参数提供了新的通用性和物理精度。通过在计算机图形学和计算光子学领域创造新的领域，项目成果有可能产生广泛的影响。其他广泛的影响将来自PI的承诺，即为未被充分代表的少数民族促进STEM教育，以及促进UCI在本科和高中阶段的推广计划，包括实验室访问，以获得软件开发的实践经验。这项研究将使可微分和逆渲染是高效的，物理上准确的，并且足够通用，以处理各种光传输现象下的任意场景参数。这项工作将做出以下四个核心贡献：首先，设计新的数学工具来描述虚拟场景中无穷小的变化如何影响光的分布，支持各种光传输模型，包括稳态、偏振和瞬态；其次，引入基于物理的可微渲染算法，在提供低方差导数估计的同时，享受新公式的通用性；第三，利用这些算法构建能够有效处理复杂现实世界配置的可微分渲染软件系统；最后，利用新的算法和系统引入基于物理的反向渲染管道，提供新的通用性和准确性，使广泛的应用受益。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project will develop new computational tools to infer physical parameters such as object shape and optical properties of a scene from measured images such as photographs. These tools are essential for building digital twins of real-world objects and will enable new applications in a wide array of fields including computer vision, computational imaging, robotics, and virtual/augmented reality (VR/AR). Unlike many existing methods that are purely data-driven, this research will develop inference techniques that leverage a simulator of how light propagates. This simulator will be differentiable, meaning that it is possible to smoothly relate its control parameters to its decisions, offering a new level of generality and physical accuracy for recovering parameters reliably under complex scenarios such as illumination from reflected light. Project outcomes have the potential for broad impact by creating new areas in computer graphics and computational photonics. Additional broad impact will derive from the PI's commitment to promoting STEM education for underrepresented minorities, and from the project facilitating UCI’s outreach programs at the undergraduate and high school levels including lab visits to allow hands-on experience with software development. This research will enable differentiable and inverse rendering that is efficient, physically accurate, and sufficiently general to handle arbitrary scene parameters under a wide variety of light transport phenomena. The work will make the following four core contributions: first, devising new mathematical tools to describe how infinitesimal changes in a virtual scene affect the distribution of light, supporting a variety of light transport models including steady state, polarized, and transient; second, introducing physics-based differentiable rendering algorithms that enjoy the generality of the new formulations while providing low-variance derivative estimates; third, leveraging these algorithms to build differentiable rendering software systems capable of efficiently handling complex real-world configurations; and, lastly, utilizing the new algorithms and systems to introduce physics-based inverse rendering pipelines that offer a new level of generality and accuracy benefiting a wide array of applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.