Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes
Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes
复制标题
DOI：
10.1109/cvpr46437.2021.00782
复制
发表时间：
2021-04
期刊：
影响因子：
--
通讯作者：
Julian Chibane;Aayush Bansal;Verica Lazova;Gerard Pons-Moll
中科院分区：
文献类型：
--
作者：
Julian Chibane;Aayush Bansal;Verica Lazova;Gerard Pons-Moll
最近的神经视图合成方法已经取得了令人印象深刻的质量和真实感，超越了传统的依赖于多视图重建的管道。最先进的方法，如NeRF[34]，旨在通过神经网络学习单个场景，并需要密集的多视图输入。在新场景上进行测试需要从头开始重新训练，这需要2-3天。在这项工作中，我们引入了立体辐射场（SRF），这是一种端到端训练的神经视图合成方法，可以推广到新的场景，并且在测试时只需要稀疏的视图。该方法的核心思想是受经典多视图立体方法启发的神经结构，通过在立体图像中寻找相似的图像区域来估计表面点。在SRF中，我们预测每个3D点的颜色和密度，给定其在输入图像中的立体对应编码。编码是通过成对相似性的集合来隐式学习的——模仿经典立体声。实验表明，SRF在场景上学习结构，而不是过度拟合。我们在DTU数据集的多个场景上进行训练，并在不重新训练的情况下推广到新的场景，只需要10个稀疏和展开的视图作为输入。我们表明，10-15分钟的微调可以进一步改善结果，获得比特定场景模型更清晰、更详细的结果。代码、模型和视频可从https://virtualhumans.mpi-inf.mpg.de/srf/获得。
Recent neural view synthesis methods have achieved impressive quality and realism, surpassing classical pipelines which rely on multi-view reconstruction. State-of-the-Art methods, such as NeRF [34], are designed to learn a single scene with a neural network and require dense multi-view inputs. Testing on a new scene requires re-training from scratch, which takes 2-3 days. In this work, we introduce Stereo Radiance Fields (SRF), a neural view synthesis approach that is trained end-to-end, generalizes to new scenes, and requires only sparse views at test time. The core idea is a neural architecture inspired by classical multi-view stereo methods, which estimates surface points by finding similar image regions in stereo images. In SRF, we predict color and density for each 3D point given an encoding of its stereo correspondence in the input images. The encoding is implicitly learned by an ensemble of pair-wise similarities – emulating classical stereo. Experiments show that SRF learns structure instead of over-fitting on a scene. We train on multiple scenes of the DTU dataset and generalize to new ones without re-training, requiring only 10 sparse and spread-out views as input. We show that 10-15 minutes of fine-tuning further improve the results, achieving significantly sharper, more detailed results than scene-specific models. The code, model, and videos are available – https://virtualhumans.mpi-inf.mpg.de/srf/.