Multi-Camera Lighting Estimation for Photorealistic Front-Facing Mobile Augmented Reality
Multi-Camera Lighting Estimation for Photorealistic Front-Facing Mobile Augmented Reality
复制标题
DOI：
10.1145/3572864.3580337
复制
发表时间：
2023-01
期刊：
影响因子：
--
通讯作者：
Yiqin Zhao;S. Fanello;Tian Guo
中科院分区：
文献类型：
--
作者：
Yiqin Zhao;S. Fanello;Tian Guo
在虚拟物体合成中，包括在移动增强现实（AR）应用中，对光照的理解起着重要作用。先前的工作通常旨在从物理环境中恢复光照以支持逼真的AR渲染。由于常见的工作流程是使用后置摄像头捕捉物理世界以便叠加虚拟物体，我们将这种使用模式称为后置AR。然而，现有方法在支持新兴的前置移动AR应用方面往往存在不足，例如虚拟试戴，用户利用前置摄像头来探索不同风格的各种产品（如眼镜或帽子）的效果。这种支持的缺乏可归因于从前置摄像头和现有技术获取360°高动态范围（HDR）环境图（一种理想的光照表示形式）所面临的独特挑战。在本文中，我们提议利用双摄像头流，通过结合多视图光照重建和参数化方向光照估计来生成高质量的环境图。我们的初步结果表明，与商业解决方案相比，使用双摄像头设置进行前置AR可提高渲染质量。
Lighting understanding plays an important role in virtual object composition, including mobile augmented reality (AR) applications. Prior work often targets recovering lighting from the physical environment to support photorealistic AR rendering. Because the common workflow is to use a back-facing camera to capture the physical world for overlaying virtual objects, we refer to this usage pattern as back-facing AR. However, existing methods often fall short in supporting emerging front-facing mobile AR applications, e.g., virtual try-on where a user leverages a front-facing camera to explore the effect of various products (e.g., glasses or hats) of different styles. This lack of support can be attributed to the unique challenges of obtaining 360° HDR environment maps, an ideal format of lighting representation, from the front-facing camera and existing techniques. In this paper, we propose to leverage dual-camera streaming to generate a high-quality environment map by combining multi-view lighting reconstruction and parametric directional lighting estimation. Our preliminary results show improved rendering quality using a dual-camera setup for front-facing AR compared to a commercial solution.