MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesis
MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesis
复制标题
DOI：
10.1109/iccv48922.2021.01235
复制
发表时间：
2021-03
期刊：
影响因子：
--
通讯作者：
Jiaxin Li;Zijian Feng;Qi She;Henghui Ding;Changhu Wang;G. Lee
中科院分区：
文献类型：
--
作者：
Jiaxin Li;Zijian Feng;Qi She;Henghui Ding;Changhu Wang;G. Lee
在本文中，我们提出 MINE 通过单个图像的密集 3D 重建来执行新颖的视图合成和深度估计。我们的方法是通过引入神经辐射场（NeRF）对多平面图像（MPI）进行连续深度概括。给定单个图像作为输入，MINE 会预测任意深度值的 4 通道图像（RGB 和体积密度），以共同重建相机视锥体并填充被遮挡的内容。然后，可以使用可微渲染将重建和修复的视锥体轻松渲染为新颖的 RGB 或深度视图。对 RealEstate10K、KITTI 和 Flowers 光场的大量实验表明，我们的 MINE 在新颖的视图合成方面远远优于最先进的技术。我们还在 iBims-1 和 NYU-v2 上的深度估计方面取得了有竞争力的结果，而无需带注释的深度监督。我们的源代码可在 https://github.com/vincentfung13/MINE 获取。
In this paper, we propose MINE to perform novel view synthesis and depth estimation via dense 3D reconstruction from a single image. Our approach is a continuous depth generalization of the Multiplane Images (MPI) by introducing the NEural radiance fields (NeRF). Given a single image as input, MINE predicts a 4-channel image (RGB and volume density) at arbitrary depth values to jointly reconstruct the camera frustum and fill in occluded contents. The reconstructed and inpainted frustum can then be easily rendered into novel RGB or depth views using differentiable rendering. Extensive experiments on RealEstate10K, KITTI and Flowers Light Fields show that our MINE outperforms state-of-the-art by a large margin in novel view synthesis. We also achieve competitive results in depth estimation on iBims-1 and NYU-v2 without annotated depth supervision. Our source code is available at https://github.com/vincentfung13/MINE.