Physically-Based Editing of Indoor Scene Lighting from a Single Image
Physically-Based Editing of Indoor Scene Lighting from a Single Image
复制标题
DOI：
10.48550/arxiv.2205.09343
复制
发表时间：
2022-05
期刊：
影响因子：
--
通讯作者：
Zhengqin Li;Jia Shi;Sai Bi;Rui Zhu;Kalyan Sunkavalli;Milovs Havsan;Zexiang Xu;R. Ramamoorthi;Manmohan Chandraker
中科院分区：
文献类型：
--
作者：
Zhengqin Li;Jia Shi;Sai Bi;Rui Zhu;Kalyan Sunkavalli;Milovs Havsan;Zexiang Xu;R. Ramamoorthi;Manmohan Chandraker
我们提出了一种方法来编辑复杂的室内照明从一个单一的图像与其预测的深度和光源分割掩模。这是一个非常具有挑战性的问题，需要对复杂的光传输进行建模，并在仅对场景进行部分LDR观察的情况下将HDR照明与材料和几何结构分离。我们使用两个新的组件来解决这个问题：1）一个整体场景重建方法，估计场景反射率和参数化3D照明，以及2）一个神经渲染框架，根据我们的预测重新渲染场景。我们使用基于物理的室内光表示，允许直观的编辑，并推断可见和不可见光源。我们的神经渲染框架将基于物理的直接照明和阴影渲染与深度网络相结合，以近似全局照明。它可以捕捉具有挑战性的照明效果，如柔和阴影、定向照明、镜面反射材质和相互反射。以往的单幅图像逆绘制方法往往涉及到场景光照和几何，并且只支持对象插入等应用。相反，通过将参数化3D照明估计与神经场景渲染相结合，我们展示了从单个图像实现全场景重新照明的第一种自动方法，包括光源插入，移除和替换。所有源代码和数据将公开发布。
We present a method to edit complex indoor lighting from a single image with its predicted depth and light source segmentation masks. This is an extremely challenging problem that requires modeling complex light transport, and disentangling HDR lighting from material and geometry with only a partial LDR observation of the scene. We tackle this problem using two novel components: 1) a holistic scene reconstruction method that estimates scene reflectance and parametric 3D lighting, and 2) a neural rendering framework that re-renders the scene from our predictions. We use physically-based indoor light representations that allow for intuitive editing, and infer both visible and invisible light sources. Our neural rendering framework combines physically-based direct illumination and shadow rendering with deep networks to approximate global illumination. It can capture challenging lighting effects, such as soft shadows, directional lighting, specular materials, and interreflections. Previous single image inverse rendering methods usually entangle scene lighting and geometry and only support applications like object insertion. Instead, by combining parametric 3D lighting estimation with neural scene rendering, we demonstrate the first automatic method to achieve full scene relighting, including light source insertion, removal, and replacement, from a single image. All source code and data will be publicly released.