Deep blending for free-viewpoint image-based rendering
Deep blending for free-viewpoint image-based rendering
复制标题
DOI：
10.1145/3272127.3275084
复制
发表时间：
2018-12
期刊：
影响因子：
--
通讯作者：
Peter Hedman;J. Philip;True Price;Jan-Michael Frahm;G. Drettakis;G. Brostow
中科院分区：
文献类型：
--
作者：
Peter Hedman;J. Philip;True Price;Jan-Michael Frahm;G. Drettakis;G. Brostow
基于图像的免费观看渲染（IBR）是一个坚定的挑战。 IBR方法将输入照片的扭曲版本结合在一起，以合成一种新颖的视图。这种组合的图像质量直接受到多视图立体（MVS）重建的几何不正确的影响，并受到视图和图像依赖性效果的影响，这些效果在混合不同输入视图的贡献时会产生伪影。我们提出了一种新的深度学习方法来融合IBR，其中我们使用持有的真实图像数据来学习混合权重以结合输入照片贡献。我们的深层混合方法要求我们应对几个挑战，以实现交互式免费视图IBR导航的目标。我们首先需要提供足够准确的几何形状，以便卷积神经网络（CNN）可以成功找到正确的混合权重。我们通过将两种不同的MV重建与互补的准确性与完整性权衡相结合来做到这一点。要将学习紧密整合到交互式IBR系统中，我们需要调整渲染算法以产生固定数量的输入层，然后可以通过CNN将其混合。我们使用每个输入照片作为持有方法中的每个输入照片来生成各种捕获的场景的培训数据。我们还设计了网络体系结构和训练损失，以提供高质量的新型视图综合，同时减少时间闪烁的工件。我们的结果表明，在各种场景中的自由视图IBR，显然超过了以前的视觉质量方法，尤其是在远离输入摄像机的远处时。
Free-viewpoint image-based rendering (IBR) is a standing challenge. IBR methods combine warped versions of input photos to synthesize a novel view. The image quality of this combination is directly affected by geometric inaccuracies of multi-view stereo (MVS) reconstruction and by view- and image-dependent effects that produce artifacts when contributions from different input views are blended. We present a new deep learning approach to blending for IBR, in which we use held-out real image data to learn blending weights to combine input photo contributions. Our Deep Blending method requires us to address several challenges to achieve our goal of interactive free-viewpoint IBR navigation. We first need to provide sufficiently accurate geometry so the Convolutional Neural Network (CNN) can succeed in finding correct blending weights. We do this by combining two different MVS reconstructions with complementary accuracy vs. completeness tradeoffs. To tightly integrate learning in an interactive IBR system, we need to adapt our rendering algorithm to produce a fixed number of input layers that can then be blended by the CNN. We generate training data with a variety of captured scenes, using each input photo as ground truth in a held-out approach. We also design the network architecture and the training loss to provide high quality novel view synthesis, while reducing temporal flickering artifacts. Our results demonstrate free-viewpoint IBR in a wide variety of scenes, clearly surpassing previous methods in visual quality, especially when moving far from the input cameras.