HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling
HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling
复制标题
DOI：
10.1109/cvpr52729.2023.01594
复制
发表时间：
2023-01
期刊：
影响因子：
--
通讯作者：
Benjamin Attal;Jia-Bin Huang;Christian Richardt;Michael Zollhoefer;J. Kopf;Matthew O’Toole;Changil Kim
中科院分区：
文献类型：
--
作者：
Benjamin Attal;Jia-Bin Huang;Christian Richardt;Michael Zollhoefer;J. Kopf;Matthew O’Toole;Changil Kim
体场景表示使照片级真实感视图合成的静态场景，并形成现有的几个6自由度视频技术的基础。然而，驱动这些表示的体绘制过程需要在质量、绘制速度和存储器效率方面进行仔细的权衡。特别是，现有的方法无法同时实现实时性能，小内存占用，和高质量的渲染具有挑战性的现实世界的场景。为了解决这些问题，我们提出了HyperReel-一种新颖的6自由度视频表示。HyperReel的两个核心组件是：（1）光线调节的样本预测网络，可在高分辨率下实现高保真、高帧率渲染;（2）紧凑且内存高效的动态体积表示。我们的6-DoF视频管道在视觉质量方面实现了与现有和当代方法相比的最佳性能，内存需求小，同时在百万像素分辨率下以高达18帧/秒的速度渲染，而无需任何自定义CUDA代码。
Volumetric scene representations enable photorealistic view synthesis for static scenes and form the basis of several existing 6-DoF video techniques. However, the volume rendering procedures that drive these representations necessitate careful trade-offs in terms of quality, rendering speed, and memory efficiency. In particular, existing methods fail to simultaneously achieve real-time performance, small memory footprint, and high-quality rendering for challenging real-world scenes. To address these issues, we present HyperReel―a novel 6-DoF video representation. The two core components of HyperReel are: (1) a ray-conditioned sample prediction network that enables high-fidelity, high frame rate rendering at high resolutions and (2) a compact and memory-efficient dynamic volume representation. Our 6-DoF video pipeline achieves the best performance compared to prior and contemporary approaches in terms of visual quality with small memory requirements, while also rendering at up to 18 frames-per-second at megapixel resolution without any custom CUDA code.