High-fidelity facial reflectance and geometry inference from an unconstrained image
High-fidelity facial reflectance and geometry inference from an unconstrained image
复制标题
DOI：
10.1145/3197517.3201364
复制
发表时间：
2018-07
期刊：
影响因子：
--
通讯作者：
Shugo Yamaguchi;Shunsuke Saito;Koki Nagano;Yajie Zhao;Weikai Chen;Kyle Olszewski;S. Morishima;Hao Li
中科院分区：
文献类型：
--
作者：
Shugo Yamaguchi;Shunsuke Saito;Koki Nagano;Yajie Zhao;Weikai Chen;Kyle Olszewski;S. Morishima;Hao Li
我们提出了一种基于深度学习的技术来推断高质量的面部反射率和几何形状，给定受试者的单个无约束图像，其中可能包含部分遮挡和任意照明条件。重建的高分辨率纹理，这是在短短几秒钟内生成的，包括高分辨率的皮肤表面反射率地图，同时表示漫反射和镜面反射，以及中频和高频位移地图，从而使我们能够在新的照明条件下呈现引人注目的数字化身。为了提取这些数据，我们使用高质量的皮肤反射率和几何数据库来训练我们的深度神经网络，该数据库是使用偏振梯度照明的最先进的多视图光度立体系统创建的。给定从输入图像中提取的原始面部纹理图，我们的神经网络合成完整的反射和位移图，以及由遮挡引起的完整缺失区域。由于我们的网络架构，完成的纹理在整个面部表现出一致的质量，该网络架构从可见区域传播纹理特征，从而产生与可见区域一致的高保真细节。我们描述了如何通过将完整的推理划分为较小的任务，并通过专用的神经网络来解决这个高度欠约束的问题。我们证明了我们的网络设计的有效性与强大的纹理完成的图像，在很大程度上被遮挡的脸。与推断的反射率和几何数据，我们演示了渲染高保真的3D化身从不同的照明条件下捕获的各种主题。此外，我们进行的评估表明，我们的方法可以推断出合理的面部反射率和几何细节相比，从高端捕捉设备，并优于替代方法，只需要一个单一的无约束的输入图像。
We present a deep learning-based technique to infer high-quality facial reflectance and geometry given a single unconstrained image of the subject, which may contain partial occlusions and arbitrary illumination conditions. The reconstructed high-resolution textures, which are generated in only a few seconds, include high-resolution skin surface reflectance maps, representing both the diffuse and specular albedo, and medium- and high-frequency displacement maps, thereby allowing us to render compelling digital avatars under novel lighting conditions. To extract this data, we train our deep neural networks with a high-quality skin reflectance and geometry database created with a state-of-the-art multi-view photometric stereo system using polarized gradient illumination. Given the raw facial texture map extracted from the input image, our neural networks synthesize complete reflectance and displacement maps, as well as complete missing regions caused by occlusions. The completed textures exhibit consistent quality throughout the face due to our network architecture, which propagates texture features from the visible region, resulting in high-fidelity details that are consistent with those seen in visible regions. We describe how this highly underconstrained problem is made tractable by dividing the full inference into smaller tasks, which are addressed by dedicated neural networks. We demonstrate the effectiveness of our network design with robust texture completion from images of faces that are largely occluded. With the inferred reflectance and geometry data, we demonstrate the rendering of high-fidelity 3D avatars from a variety of subjects captured under different lighting conditions. In addition, we perform evaluations demonstrating that our method can infer plausible facial reflectance and geometric details comparable to those obtained from high-end capture devices, and outperform alternative approaches that require only a single unconstrained input image.