DyLiN: Making Light Field Networks Dynamic
DyLiN: Making Light Field Networks Dynamic
复制标题
DOI：
10.1109/cvpr52729.2023.01193
复制
发表时间：
2023-03
期刊：
影响因子：
--
通讯作者：
Heng Yu;Joel Julin;Z. '. Milacski;Koichiro Niinuma;László A. Jeni
中科院分区：
文献类型：
--
作者：
Heng Yu;Joel Julin;Z. '. Milacski;Koichiro Niinuma;László A. Jeni
光场网络是辐射场到定向射线的重新表述，其速度比其坐标网络对应物快，并且在从2D观测表示3D结构方面提供更高的保真度。它们非常适合通用场景表示和操作，但有一个问题：它们仅限于整体和静态场景。在本文中，我们提出了动态光场网络（DyLiN）方法，可以处理非刚性变形，包括拓扑变化。我们学习从输入射线到正则射线的变形场，并将它们提升到更高维的空间以处理不连续性。我们进一步介绍CoDyLiN，它增强了DyLiN与可控属性输入。我们通过从预先训练的动态辐射场中提取知识来训练这两个模型。我们使用合成和真实的世界数据集，包括各种非刚性变形的DyLiN进行评估。DyLiN在视觉保真度方面定性优于和定量匹配最先进的方法，同时计算速度快25 - 71倍。我们还在属性注释数据上测试了CoDyLiN，它超过了它的教师模型。项目页面：https://dylin2023.github.io。
Light Field Networks, the re-formulations of radiance fields to oriented rays, are magnitudes faster than their coordinate network counterparts, and provide higher fidelity with respect to representing 3D structures from 2D observations. They would be well suited for generic scene representation and manipulation, but suffer from one problem: they are limited to holistic and static scenes. In this paper, we propose the Dynamic Light Field Network (DyLiN) method that can handle non-rigid deformations, including topological changes. We learn a deformation field from input rays to canonical rays, and lift them into a higher dimensional space to handle discontinuities. We further introduce CoDyLiN, which augments DyLiN with controllable attribute inputs. We train both models via knowledge distillation from pretrained dynamic radiance fields. We evaluated DyLiN using both synthetic and real world datasets that include various non-rigid deformations. DyLiN qualitatively outperformed and quantitatively matched state-of-the-art methods in terms of visual fidelity, while being 25 – 71× computationally faster. We also tested CoDyLiN on attribute annotated data and it surpassed its teacher model. Project page: https://dylin2023.github.io.