Towards Learning Neural Representations from Shadows
Towards Learning Neural Representations from Shadows
复制标题
从阴影中学习神经表征
DOI：
--
复制
发表时间：
2022
期刊：
影响因子：
--
通讯作者：
R. Raskar
中科院分区：
文献类型：
--
作者：
Kushagra Tiwary;Tzofi Klinghoffer;R. Raskar
我们提出了一种学习神经影场的方法，神经影场是仅从场景中存在的阴影中学习的神经场景表示。虽然传统的阴影形状(SFS)算法从阴影重建几何图形，但它们假设了固定的扫描设置，无法推广到复杂场景。另一方面，神经渲染算法依赖于RGB图像之间的光度学一致性，但在很大程度上忽略了阴影等物理线索，这些线索已被证明提供了有关场景的有价值的信息。我们观察到，阴影是一种强大的线索，可以约束神经场景表征来学习SFS，甚至优于NERF来重建原本隐藏的几何图形。我们提出了一种受图形启发的可微化方法，通过体绘制来绘制准确的阴影，预测出可以与地面真实阴影相比较的阴影贴图。即使只使用二进制阴影贴图，我们也表明神经渲染可以定位对象并估计粗略的几何图形。我们的方法揭示了图像中的稀疏线索可以用来使用可微分体绘制来估计几何图形。此外，我们的框架具有很高的通用性，可以与现有的仅使用光度一致性的3D重建技术一起工作。
We present a method that learns neural shadow fields which are neural scene representations that are only learnt from the shadows present in the scene. While traditional shape-from-shadow (SfS) algorithms reconstruct geometry from shadows, they assume a fixed scanning setup and fail to generalize to complex scenes. Neural rendering algorithms, on the other hand, rely on photometric consistency between RGB images, but largely ignore physical cues such as shadows, which have been shown to provide valuable information about the scene. We observe that shadows are a powerful cue that can constrain neural scene representations to learn SfS, and even outperform NeRF to reconstruct otherwise hidden geometry. We propose a graphics-inspired differentiable approach to render accurate shadows with volumetric rendering, predicting a shadow map that can be compared to the ground truth shadow. Even with just binary shadow maps, we show that neural rendering can localize the object and estimate coarse geometry. Our approach reveals that sparse cues in images can be used to estimate geometry using differentiable volumetric rendering. Moreover, our framework is highly generalizable and can work alongside existing 3D reconstruction techniques that otherwise only use photometric consistency.
DOI：
--
发表时间：
2021-10
期刊：
--
影响因子：
--
作者：
Jason Y. Zhang;Gengshan Yang;Shubham Tulsiani;Deva Ramanan
通讯作者：
Jason Y. Zhang;Gengshan Yang;Shubham Tulsiani;Deva Ramanan