Neural Kaleidoscopic Space Sculpting
Neural Kaleidoscopic Space Sculpting
复制标题
DOI：
10.1109/cvpr52729.2023.00423
复制
发表时间：
2023-06
期刊：
影响因子：
--
通讯作者：
Byeongjoo Ahn;Michael De Zeeuw;Ioannis Gkioulekas;Aswin C. Sankaranarayanan
中科院分区：
文献类型：
--
作者：
Byeongjoo Ahn;Michael De Zeeuw;Ioannis Gkioulekas;Aswin C. Sankaranarayanan
我们介绍了一种方法，从一个单一的万花筒图像使用神经表面表示恢复全环绕三维重建。全环绕3D重建对于许多应用至关重要，例如增强现实和虚拟现实。万花筒使用单个摄像头和多个镜子，是实现全环绕覆盖的一种方便方法，因为它重新分配光线方向，从而在单个图像中捕获多个视点。这实现了单次拍摄和动态全环绕3D重建。然而，将万花筒图像用于多视图立体是具有挑战性的，因为我们需要通过识别哪个像素对应于哪个虚拟相机来将图像分解为多视图图像，我们称之为标记的过程。为了解决这一挑战，pur方法避免了显式估计标签的需要，而是通过仔细使用万花筒图像中存在的轮廓，背景，前景和纹理信息来“雕刻”神经表面表示。我们证明了我们的方法在一系列的模拟和真实的实验，静态和动态场景的优势。
We introduce a method that recovers full-surround 3D reconstructions from a single kaleidoscopic image using a neural surface representation. Full-surround 3D reconstruction is critical for many applications, such as augmented and virtual reality. A kaleidoscope, which uses a single camera and multiple mirrors, is a convenient way of achieving full-surround coverage, as it redistributes light directions and thus captures multiple viewpoints in a single image. This enables single-shot and dynamic full-surround 3D reconstruction. However, using a kaleidoscopic image for multiview stereo is challenging, as we need to decompose the image into multi-view images by identifying which pixel corresponds to which virtual camera, a process we call labeling. To address this challenge, pur approach avoids the need to explicitly estimate labels, but instead “sculpts” a neural surface representation through the careful use of silhouette, background, foreground, and texture information present in the kaleidoscopic image. We demonstrate the advantages of our method in a range of simulated and real experiments, on both static and dynamic scenes.