HyP-NeRF: Learning Improved NeRF Priors using a HyperNetwork
HyP-NeRF: Learning Improved NeRF Priors using a HyperNetwork
复制标题
DOI：
10.48550/arxiv.2306.06093
复制
发表时间：
2023-06
期刊：
影响因子：
--
通讯作者：
Bipasha Sen;Gaurav Singh;Aditya Agarwal;Rohith Agaram;K. Krishna;Srinath Sridhar
中科院分区：
文献类型：
--
作者：
Bipasha Sen;Gaurav Singh;Aditya Agarwal;Rohith Agaram;K. Krishna;Srinath Sridhar
神经辐射场（NeRF）已经成为一种越来越流行的表示，以捕捉高质量的外观和形状的场景和对象。然而，由于网络权重空间的高维性，学习场景或对象类别的可泛化NeRF先验一直具有挑战性。为了解决现有工作在泛化、多视图一致性和提高质量方面的局限性，我们提出了一种使用超网络学习可泛化类别级NeRF先验的潜在条件反射方法hypf -NeRF。我们不是使用超网络来估计NeRF的权重，而是同时估计权重和多分辨率哈希编码，从而显著提高质量。为了进一步提高质量，我们采用了一种降噪和微调策略，该策略对由超网络估计的nerf渲染的图像进行降噪，并在保持多视图一致性的同时对其进行微调。这些改进使我们能够使用HyP-NeRF作为多个下游任务的通用先验，包括从单视图或混乱场景和文本到NeRF的NeRF重建。我们对hypnerf的三个任务进行了定性比较和评估：泛化、压缩和检索，展示了我们最先进的结果。
Neural Radiance Fields (NeRF) have become an increasingly popular representation to capture high-quality appearance and shape of scenes and objects. However, learning generalizable NeRF priors over categories of scenes or objects has been challenging due to the high dimensionality of network weight space. To address the limitations of existing work on generalization, multi-view consistency and to improve quality, we propose HyP-NeRF, a latent conditioning method for learning generalizable category-level NeRF priors using hypernetworks. Rather than using hypernetworks to estimate only the weights of a NeRF, we estimate both the weights and the multi-resolution hash encodings resulting in significant quality gains. To improve quality even further, we incorporate a denoise and finetune strategy that denoises images rendered from NeRFs estimated by the hypernetwork and finetunes it while retaining multiview consistency. These improvements enable us to use HyP-NeRF as a generalizable prior for multiple downstream tasks including NeRF reconstruction from single-view or cluttered scenes and text-to-NeRF. We provide qualitative comparisons and evaluate HyP-NeRF on three tasks: generalization, compression, and retrieval, demonstrating our state-of-the-art results.