pixelNeRF: Neural Radiance Fields from One or Few Images
pixelNeRF: Neural Radiance Fields from One or Few Images
复制标题
DOI：
10.1109/cvpr46437.2021.00455
复制
发表时间：
2020-12
期刊：
影响因子：
--
通讯作者：
Alex Yu;Vickie Ye;Matthew Tancik;Angjoo Kanazawa
中科院分区：
文献类型：
--
作者：
Alex Yu;Vickie Ye;Matthew Tancik;Angjoo Kanazawa
我们提出了pixelNeRF，这是一个学习框架，可以预测基于一个或几个输入图像的连续神经场景表示。现有的构建神经辐射场[27]的方法涉及对每个场景的独立优化表示，需要许多校准视图和大量的计算时间。我们通过引入一种架构来解决这些缺点，该架构以完全卷积的方式对图像输入进行NeRF。这使得网络可以跨多个场景进行训练，提前学习一个场景，使其能够从稀疏的视图集（少到一个）以前馈方式执行新的视图合成。利用NeRF的体绘制方法，我们的模型可以直接从图像中进行训练，而无需明确的3D监督。我们在ShapeNet基准上进行了广泛的实验，用于单幅图像的新视图合成任务，包括持有的对象以及整个看不见的类别。通过在多目标ShapeNet场景和来自DTU数据集的真实场景上演示pixelNeRF，我们进一步展示了它的灵活性。在所有情况下，pixelNeRF在新视图合成和单图像3D重建方面都优于当前最先进的基线。视频和代码请访问项目网站：https://alexyu.net/pixelnerf。
We propose pixelNeRF, a learning framework that predicts a continuous neural scene representation conditioned on one or few input images. The existing approach for constructing neural radiance fields [27] involves optimizing the representation to every scene independently, requiring many calibrated views and significant compute time. We take a step towards resolving these shortcomings by introducing an architecture that conditions a NeRF on image inputs in a fully convolutional manner. This allows the network to be trained across multiple scenes to learn a scene prior, enabling it to perform novel view synthesis in a feed-forward manner from a sparse set of views (as few as one). Leveraging the volume rendering approach of NeRF, our model can be trained directly from images with no explicit 3D supervision. We conduct extensive experiments on ShapeNet benchmarks for single image novel view synthesis tasks with held-out objects as well as entire unseen categories. We further demonstrate the flexibility of pixelNeRF by demonstrating it on multi-object ShapeNet scenes and real scenes from the DTU dataset. In all cases, pixelNeRF outperforms current state-of-the-art baselines for novel view synthesis and single image 3D reconstruction. For the video and code, please visit the project website:https://alexyu.net/pixelnerf.