Instant-3D: Instant Neural Radiance Field Training Towards On-Device AR/VR 3D Reconstruction
Instant-3D: Instant Neural Radiance Field Training Towards On-Device AR/VR 3D Reconstruction
复制标题
DOI：
10.1145/3579371.3589115
复制
发表时间：
2023-04
期刊：
影响因子：
--
通讯作者：
Sixu Li;Chaojian Li;Wenbo Zhu;Bo Yu;Yang Zhao;Cheng Wan;Haoran You;Huihong Shi;Yingyan Lin
中科院分区：
文献类型：
--
作者：
Sixu Li;Chaojian Li;Wenbo Zhu;Bo Yu;Yang Zhao;Cheng Wan;Haoran You;Huihong Shi;Yingyan Lin
基于神经辐射场（NeRF）的3D重建非常适合沉浸式增强现实和虚拟现实（AR/VR）应用，但实现即时（即< 5秒）设备上的NeRF训练仍然是一个挑战。在这项工作中，我们首先确定了效率低下的瓶颈：在每次训练迭代期间，需要从3D嵌入网格中插值多达200,000次的NeRF嵌入。为了缓解这种情况，我们提出了instant - 3d，这是一种算法-硬件协同设计的加速框架，可实现设备上的即时NeRF训练。我们的算法根据颜色和密度对嵌入网格表示进行分解，通过采用不同的(1)网格大小和(2)颜色和密度分支的更新频率来压缩计算冗余。我们的硬件加速器通过(1)在前馈过程中将多个相邻点的内存读取请求映射为一个，(2)在反向传播过程中合并来自同一滑动时间窗口的嵌入网格更新，以及(3)融合不同的计算核心以支持Instant-3D算法的颜色和密度分支所需的不同网格大小，进一步减少了嵌入网格插值的主导内存访问。大量的实验验证了Instant-3D的有效性，在保持相同重建质量的情况下，将训练时间减少了41x - 248x。令人兴奋的是，instant -3D实现了AR/VR的即时3D重建，每个场景的重建时间仅为1.6秒，满足AR/VR 1.9 W的功耗限制。
Neural Radiance Field (NeRF) based 3D reconstruction is highly desirable for immersive Augmented and Virtual Reality (AR/VR) applications, but achieving instant (i.e., < 5 seconds) on-device NeRF training remains a challenge. In this work, we first identify the inefficiency bottleneck: the need to interpolate NeRF embeddings up to 200,000 times from a 3D embedding grid during each training iteration. To alleviate this, we propose Instant-3D, an algorithm-hardware co-design acceleration framework that achieves instant on-device NeRF training. Our algorithm decomposes the embedding grid representation in terms of color and density, enabling computational redundancy to be squeezed out by adopting different (1) grid sizes and (2) update frequencies for the color and density branches. Our hardware accelerator further reduces the dominant memory accesses for embedding grid interpolation by (1) mapping multiple nearby points' memory read requests into one during the feed-forward process, (2) merging embedding grid updates from the same sliding time window during back-propagation, and (3) fusing different computation cores to support the different grid sizes needed by the color and density branches of Instant-3D algorithm. Extensive experiments validate the effectiveness of Instant-3D, achieving a large training time reduction of 41× - 248× while maintaining the same reconstruction quality. Excitingly, Instant-3D has enabled instant 3D reconstruction for AR/VR, requiring a reconstruction time of only 1.6 seconds per scene and meeting the AR/VR power consumption constraint of 1.9 W.