Stable View Synthesis
Stable View Synthesis
复制标题
DOI：
10.1109/cvpr46437.2021.01204
复制
发表时间：
2020-11
期刊：
影响因子：
--
通讯作者：
Gernot Riegler;V. Koltun
中科院分区：
文献类型：
--
作者：
Gernot Riegler;V. Koltun
我们提出了稳定视图合成（SVS）。给定一组从自由分布的视点描绘场景的源图像，SVS可以合成场景的新视图。该方法在通过运动结构和多视图立体计算的几何支架上运行。这个3D支架上的每个点都与视图射线和相应的特征向量相关联，这些特征向量在输入图像中编码了该点的外观。SVS的核心是依赖于视图的表面特征聚合，其中对每个3D点的方向特征向量进行处理，为射线生成新的特征向量，将该点映射到新的目标视图中。然后通过卷积网络从以这种方式合成的所有像素的特征张量中渲染目标视图。该方法由可微模块组成，端到端训练。它支持空间变化的视相关权重和源图像在每个点的特征变换；由于表面特征聚集对目标视图的平滑依赖，空间和时间稳定性；和合成视依赖效果，如镜面反射。实验结果表明，在三个不同的现实世界数据集上，SVS在定量和定性上都优于最先进的视图合成方法，在具有挑战性的大规模场景的自由视点视频中实现了前所未有的真实感。代码可从https://github.com/intel-isl/StableViewSynthesis获得
We present Stable View Synthesis (SVS). Given a set of source images depicting a scene from freely distributed viewpoints, SVS synthesizes new views of the scene. The method operates on a geometric scaffold computed via structure-from-motion and multi-view stereo. Each point on this 3D scaffold is associated with view rays and corresponding feature vectors that encode the appearance of this point in the input images. The core of SVS is view-dependent on-surface feature aggregation, in which directional feature vectors at each 3D point are processed to produce a new feature vector for a ray that maps this point into the new target view. The target view is then rendered by a convolutional network from a tensor of features synthesized in this way for all pixels. The method is composed of differentiable modules and is trained end-to-end. It supports spatially-varying view-dependent importance weighting and feature transformation of source images at each point; spatial and temporal stability due to the smooth dependence of on-surface feature aggregation on the target view; and synthesis of view-dependent effects such as specular reflection. Experimental results demonstrate that SVS outperforms state-of-the-art view synthesis methods both quantitatively and qualitatively on three diverse real-world datasets, achieving unprecedented levels of realism in free-viewpoint video of challenging large-scale scenes. Code is available at https://github.com/intel-isl/StableViewSynthesis