FoV-NeRF: Foveated Neural Radiance Fields for Virtual Reality
FoV-NeRF: Foveated Neural Radiance Fields for Virtual Reality
复制标题
DOI：
10.1109/tvcg.2022.3203102
复制
发表时间：
2021-03
影响因子：
5.2
通讯作者：
Nianchen Deng;Zhenyi He;Jiannan Ye;Budmonde Duinkharjav;Praneeth Chakravarthula;Xubo Yang;Qi Sun
中科院分区：
文献类型：
--
作者：
Nianchen Deng;Zhenyi He;Jiannan Ye;Budmonde Duinkharjav;Praneeth Chakravarthula;Xubo Yang;Qi Sun
随着消费者显示器和商业VR平台的兴起，虚拟现实（VR）变得无处不在。这样的显示器需要低延迟和高质量的合成图像渲染，同时减少计算开销。神经渲染的最新进展表明，通过基于图像的虚拟或物理环境表示，有望在3D计算机图形学中解锁新的可能性。具体来说，神经辐射场（NeRF）证明，可以实现照片般逼真的质量和连续的视图变化的3D场景，而不会损失视图相关的效果。虽然NeRF可以为VR应用渲染带来显着的好处，但它面临着高视场，高分辨率和立体/自我中心观看所带来的独特挑战，通常会导致渲染图像的低质量和高延迟。在VR中，这不仅会损害交互体验，还可能导致疾病。为了解决这些问题，对六度的自由，自我中心，和立体NeRF在VR中，我们提出了第一个凝视的3D神经表示和视图合成方法。我们将人类心理物理学的视觉和立体视敏度到一个自我中心的神经表示的三维场景。然后，我们共同优化延迟/性能和视觉质量，同时相互桥接人类感知和神经场景合成，以实现感知高质量的沉浸式交互。我们进行了客观分析和主观研究，以评估我们的方法的有效性。我们发现，我们的方法显着降低了延迟（与NeRF相比，时间减少了99%），而不会损失高保真渲染（在感知上与全分辨率地面实况相同）。所提出的方法可以作为未来VR/AR系统的第一步，该系统可以实时捕获，传送和可视化远程环境。
Virtual Reality (VR) is becoming ubiquitous with the rise of consumer displays and commercial VR platforms. Such displays require low latency and high quality rendering of synthetic imagery with reduced compute overheads. Recent advances in neural rendering showed promise of unlocking new possibilities in 3D computer graphics via image-based representations of virtual or physical environments. Specifically, the neural radiance fields (NeRF) demonstrated that photo-realistic quality and continuous view changes of 3D scenes can be achieved without loss of view-dependent effects. While NeRF can significantly benefit rendering for VR applications, it faces unique challenges posed by high field-of-view, high resolution, and stereoscopic/egocentric viewing, typically causing low quality and high latency of the rendered images. In VR, this not only harms the interaction experience but may also cause sickness. To tackle these problems toward six-degrees-of-freedom, egocentric, and stereo NeRF in VR, we present the first gaze-contingent 3D neural representation and view synthesis method. We incorporate the human psychophysics of visual- and stereo-acuity into an egocentric neural representation of 3D scenery. We then jointly optimize the latency/performance and visual quality while mutually bridging human perception and neural scene synthesis to achieve perceptually high-quality immersive interaction. We conducted both objective analysis and subjective studies to evaluate the effectiveness of our approach. We find that our method significantly reduces latency (up to 99% time reduction compared with NeRF) without loss of high-fidelity rendering (perceptually identical to full-resolution ground truth). The presented approach may serve as the first step toward future VR/AR systems that capture, teleport, and visualize remote environments in real-time.