Relightify: Relightable 3D Faces from a Single Image via Diffusion Models
Relightify: Relightable 3D Faces from a Single Image via Diffusion Models
复制标题
DOI：
10.1109/iccv51070.2023.00809
复制
发表时间：
2023-05
期刊：
影响因子：
--
通讯作者：
Foivos Paraperas Papantoniou;Alexandros Lattas;Stylianos Moschoglou;S. Zafeiriou
中科院分区：
文献类型：
--
作者：
Foivos Paraperas Papantoniou;Alexandros Lattas;Stylianos Moschoglou;S. Zafeiriou
继扩散模型在图像生成方面取得了显着的成功之后，最近的工作也证明了它们以无监督的方式解决一些逆问题的能力，通过适当地限制基于条件输入的采样过程。出于这一动机，在本文中，我们提出了第一种方法，使用扩散模型作为先验的高精度三维面部BRDF重建从一个单一的图像。我们首先利用面部反射的高质量UV数据集（漫反射和镜面反射和法线），我们在不同的照明设置下渲染以模拟自然RGB纹理，然后在渲染纹理和反射组件的级联对上训练无条件扩散模型。在测试时，我们将3D变形模型拟合到给定图像，并在部分UV纹理中展开面部。通过从扩散模型中采样，同时保持观察到的纹理部分不变，该模型不仅可以在单个去噪步骤序列中修复自遮挡区域，还可以修复未知的反射率分量。与现有的方法相比，我们直接从输入图像中获取所观察到的纹理，从而导致更忠实和一致的反射率估计。通过一系列定性和定量的比较，我们证明了上级性能在纹理完成以及反射重建任务。
Following the remarkable success of diffusion models on image generation, recent works have also demonstrated their impressive ability to address a number of inverse problems in an unsupervised way, by properly constraining the sampling process based on a conditioning input. Motivated by this, in this paper, we present the first approach to use diffusion models as a prior for highly accurate 3D facial BRDF reconstruction from a single image. We start by leveraging a high-quality UV dataset of facial reflectance (diffuse and specular albedo and normals), which we render under varying illumination settings to simulate natural RGB textures and, then, train an unconditional diffusion model on concatenated pairs of rendered textures and reflectance components. At test time, we fit a 3D morphable model to the given image and unwrap the face in a partial UV texture. By sampling from the diffusion model, while retaining the observed texture part intact, the model inpaints not only the self-occluded areas but also the unknown reflectance components, in a single sequence of denoising steps. In contrast to existing methods, we directly acquire the observed texture from the input image, thus, resulting in more faithful and consistent reflectance estimation. Through a series of qualitative and quantitative comparisons, we demonstrate superior performance in both texture completion as well as reflectance reconstruction tasks.