One shot 3D photography
One shot 3D photography
复制标题
一键3D摄影
DOI：
10.1145/3386569.3392420
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
Michael F. Cohen
中科院分区：
文献类型：
--
作者：
J. Kopf;K. Matzen;Suhib Alsisan;Ocean Quigley;Francis Ge;Yangming Chong;Josh Patterson;Jan;Shu Wu;Matthew Yu;Peizhao Zhang;Zijian He;Péter Vajda;Ayush Saraf;Michael F. Cohen
3D 摄影是一种新媒体，可以让观看者更充分地体验捕捉到的瞬间。在这项工作中，我们将 3D 照片称为显示因移动视点而引起的视差的照片（而不是具有固定视点的立体对）。 3D 照片在时间上是静态的，就像传统照片一样，但在移动或桌面屏幕以及虚拟现实设备上以交互式视差显示，在虚拟现实设备上观看它还包括立体效果。我们提出了一个用于创建和查看 3D 照片的端到端系统，以及其中的算法和设计选择。我们的 3D 照片是单次拍摄的，并直接在移动设备上进行处理。该方法首先使用针对移动设备优化的新单目深度估计网络来估计 2D 输入图像的深度。它的性能与最先进的技术相比具有竞争力，但具有较低的延迟和峰值内存消耗，并且使用的参数少一个数量级。所得深度被提升为分层深度图像，并在视差区域中合成新的几何形状。我们还直接在 LDI 上使用修复网络（也针对移动设备进行了优化）来合成视差区域中的颜色纹理和结构。最后，我们将结果转换为基于网格的表示，即使在低端设备和较差的网络连接上也可以有效地传输和渲染。总而言之，在移动设备上处理只需几秒钟，并且可以立即查看和共享结果。我们进行广泛的定量评估来验证我们的系统并将其新组件与当前最先进的技术进行比较。
3D photography is a new medium that allows viewers to more fully experience a captured moment. In this work, we refer to a 3D photo as one that displays parallax induced by moving the viewpoint (as opposed to a stereo pair with a fixed viewpoint). 3D photos are static in time, like traditional photos, but are displayed with interactive parallax on mobile or desktop screens, as well as on Virtual Reality devices, where viewing it also includes stereo. We present an end-to-end system for creating and viewing 3D photos, and the algorithmic and design choices therein. Our 3D photos are captured in a single shot and processed directly on a mobile device. The method starts by estimating depth from the 2D input image using a new monocular depth estimation network that is optimized for mobile devices. It performs competitively to the state-of-the-art, but has lower latency and peak memory consumption and uses an order of magnitude fewer parameters. The resulting depth is lifted to a layered depth image, and new geometry is synthesized in parallax regions. We synthesize color texture and structures in the parallax regions as well, using an inpainting network, also optimized for mobile devices, on the LDI directly. Finally, we convert the result into a mesh-based representation that can be efficiently transmitted and rendered even on low-end devices and over poor network connections. Altogether, the processing takes just a few seconds on a mobile device, and the result can be instantly viewed and shared. We perform extensive quantitative evaluation to validate our system and compare its new components against the current state-of-the-art.