NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild
NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild
复制标题
DOI：
--
复制
发表时间：
2021-10
期刊：
影响因子：
--
通讯作者：
Jason Y. Zhang;Gengshan Yang;Shubham Tulsiani;Deva Ramanan
中科院分区：
文献类型：
--
作者：
Jason Y. Zhang;Gengshan Yang;Shubham Tulsiani;Deva Ramanan
近年来，探索几何和辐射的隐式表示的工作有了巨大的增长，并通过神经辐射场（NeRF）推广。这些作品基本上是基于一个（隐式）体积表示的占用，使他们能够模拟不同的场景结构，包括半透明的对象和大气遮蔽。但是，由于绝大多数真实世界的场景是由定义良好的表面，我们引入了一个表面模拟这样的隐式模型称为神经反射表面（NeRS）。NeRS学习一个封闭曲面的神经形状表示，该曲面与球体同构，保证了水密重建。更重要的是，表面参数化允许NeRS学习（神经）双向表面反射函数（BRDF），将视图相关外观分解为环境照明，漫射颜色（漫射）和镜面反射。“最后，我们不是在合成场景或实验室控制的捕获上说明我们的结果，而是从在线市场收集了一个新的多视图图像数据集，用于销售商品。这样的“野外“多视图图像集带来了许多挑战，包括具有未知/粗略相机估计的少量视图。我们证明了基于表面的神经重建能够从这些数据中学习，优于基于体积神经渲染的重建。我们希望NeRS作为构建可扩展的，高质量的真实世界形状，材料和照明库的第一步。包含代码和视频可视化的项目页面可以在https://jasonyzhang.com/ners上找到。
Recent history has seen a tremendous growth of work exploring implicit representations of geometry and radiance, popularized through Neural Radiance Fields (NeRF). Such works are fundamentally based on a (implicit) volumetric representation of occupancy, allowing them to model diverse scene structure including translucent objects and atmospheric obscurants. But because the vast majority of real-world scenes are composed of well-defined surfaces, we introduce a surface analog of such implicit models called Neural Reflectance Surfaces (NeRS). NeRS learns a neural shape representation of a closed surface that is diffeomorphic to a sphere, guaranteeing water-tight reconstructions. Even more importantly, surface parameterizations allow NeRS to learn (neural) bidirectional surface reflectance functions (BRDFs) that factorize view-dependent appearance into environmental illumination, diffuse color (albedo), and specular"shininess."Finally, rather than illustrating our results on synthetic scenes or controlled in-the-lab capture, we assemble a novel dataset of multi-view images from online marketplaces for selling goods. Such"in-the-wild"multi-view image sets pose a number of challenges, including a small number of views with unknown/rough camera estimates. We demonstrate that surface-based neural reconstructions enable learning from such data, outperforming volumetric neural rendering-based reconstructions. We hope that NeRS serves as a first step toward building scalable, high-quality libraries of real-world shape, materials, and illumination. The project page with code and video visualizations can be found at https://jasonyzhang.com/ners.