DeProCams: Simultaneous Relighting, Compensation and Shape Reconstruction for Projector-Camera Systems
DeProCams: Simultaneous Relighting, Compensation and Shape Reconstruction for Projector-Camera Systems
复制标题
DOI：
10.1109/tvcg.2021.3067771
复制
发表时间：
2021-03
影响因子：
5.2
通讯作者：
Bingyao Huang;Haibin Ling
中科院分区：
文献类型：
--
作者：
Bingyao Huang;Haibin Ling
基于图像的重光照、投影补偿和深度/法线重建是投影-相机系统（ProCams）和空间增强现实（SAR）的三个重要任务。虽然它们共享一个相似的寻找投影仪-摄像机图像映射的管道，但在传统上，它们是独立处理的，有时有不同的先决条件、设备和采样图像。在实践中，这对于SAR应用程序逐个处理它们可能很麻烦。在本文中，我们提出了一种新的端到端可训练模型DeProCams来明确学习ProCams的光度和几何映射，并且一旦训练完成，DeProCams可以同时应用于这三个任务。DeProCams明确地将投影机-相机图像映射分解为三个子过程：阴影属性估计、粗糙的直接光线估计和逼真的神经渲染。DeProCams解决的一个特殊挑战是遮挡，为此我们利用极面约束并提出了一种新的可微投影仪直射光罩。因此，它可以与其他模块一起端到端学习。然后，为了提高收敛性，我们应用了光度和几何约束，使中间结果是可信的。在我们的实验中，DeProCams比以前的艺术表现出明显的优势，具有良好的质量，同时具有完全的可微分性。此外，通过在一个统一的模型中解决这三个任务，DeProCams放弃了对额外光学设备、辐射校准和结构光的需求。
Image-based relighting, projector compensation and depth/normal reconstruction are three important tasks of projector-camera systems (ProCams) and spatial augmented reality (SAR). Although they share a similar pipeline of finding projector-camera image mappings, in tradition, they are addressed independently, sometimes with different prerequisites, devices and sampling images. In practice, this may be cumbersome for SAR applications to address them one-by-one. In this paper, we propose a novel end-to-end trainable model named DeProCams to explicitly learn the photometric and geometric mappings of ProCams, and once trained, DeProCams can be applied simultaneously to the three tasks. DeProCams explicitly decomposes the projector-camera image mappings into three subprocesses: shading attributes estimation, rough direct light estimation and photorealistic neural rendering. A particular challenge addressed by DeProCams is occlusion, for which we exploit epipolar constraint and propose a novel differentiable projector direct light mask. Thus, it can be learned end-to-end along with the other modules. Afterwards, to improve convergence, we apply photometric and geometric constraints such that the intermediate results are plausible. In our experiments, DeProCams shows clear advantages over previous arts with promising quality and meanwhile being fully differentiable. Moreover, by solving the three tasks in a unified model, DeProCams waives the need for additional optical devices, radiometric calibrations and structured light.