Differentiable Appearance Acquisition from a Flash/No-flash RGB-D Pair
Differentiable Appearance Acquisition from a Flash/No-flash RGB-D Pair
复制标题
DOI：
10.1109/iccp54855.2022.9887646
复制
发表时间：
2022-08
期刊：
影响因子：
--
通讯作者：
Hyunjin Ku;Hyunho Hat;J. Lee;Dahyun Kang;J. Tompkin;Min H. Kim
中科院分区：
文献类型：
--
作者：
Hyunjin Ku;Hyunho Hat;J. Lee;Dahyun Kang;J. Tompkin;Min H. Kim
在自然环境中重建3D物体需要解决几何、空间变化材料和光照估计的不适定问题。因此，许多方法不切实际地限制于黑暗环境，使用受控照明装备，或者使用很少的手持捕获但遭受降低的质量。我们开发了一种方法，该方法仅使用在环境照明中捕获的两次智能手机曝光来重建外观，比基线方法更准确和实用。我们的见解是，我们可以使用闪光灯/无闪光灯RGB-D对来使用点照明提出逆渲染问题。这允许高效的可微分渲染，以从良好的初始化优化深度和法线，并且还同时优化漫射环境照明和SVBRDF材料。我们发现，这减少了25%的漫反射误差，镜面反射误差46%，正常误差30%，对单一和成对的图像基线，使用基于学习的技术。鉴于我们的方法对于日常固体物体是实用的，我们可以为移动的摄影提供逼真的重新照明，并为增强现实提供更轻松的内容创建。
Reconstructing 3D objects in natural environments requires solving the ill-posed problem of geometry, spatially-varying material, and lighting estimation. As such, many approaches impractically constrain to a dark environment, use controlled lighting rigs, or use few handheld captures but suffer reduced quality. We develop a method that uses just two smartphone exposures captured in ambient lighting to reconstruct appearance more accurately and practically than baseline methods. Our insight is that we can use a flash/no-flash RGB-D pair to pose an inverse rendering problem using point lighting. This allows efficient differentiable rendering to optimize depth and normals from a good initialization and so also the simultaneous optimization of diffuse environment illumination and SVBRDF material. We find that this reduces diffuse albedo error by 25%, specular error by 46%, and normal error by 30% against single-and paired-image baselines that use learning-based techniques. Given that our approach is practical for everyday solid objects, we enable photorealistic relighting for mobile photography and easier content creation for augmented reality.