A Unified Framework for High Fidelity Face Swap and Expression Reenactment
A Unified Framework for High Fidelity Face Swap and Expression Reenactment
复制标题
高保真面部交换和表情重演的统一框架
DOI：
10.1109/tcsvt.2021.3106047
复制
发表时间：
2022-06-01
影响因子：
8.4
通讯作者：
Lyu, Siwei
中科院分区：
文献类型：
--
作者：
Peng, Bo;Fan, Hongxing;Lyu, Siwei
随着强大的图像生成模型的发展，人脸操作技术得到了快速的发展。两种特殊的人脸操作方法，即人脸交换和表情再现，由于其灵活性和容易产生高质量的合成结果而备受关注。最近，这两个主题正在积极研究。然而，大多数现有的方法将这两个任务分开处理，忽略了它们潜在的相似性。在本文中，我们建议在一个统一的框架内，实现高质量的合成结果来解决这两个问题。我们的统一框架的使能组件是3D姿态，形状和表达因素的清晰解缠，然后相应地将它们重新组合用于不同的任务。然后，我们使用相同的一组2D表示进行面部交换和表情再现任务，这些任务被输入到一个通用的图像转换模型中，以直接生成最终的合成图像。一旦训练，该模型可以完成面部交换和表情再现任务，以前看不见的主题。综合实验和比较表明，该方法在多个方面都取得了较高的逼真度，尤其是在人脸交换任务中能够忠实地保持源人脸形状，在表情再现任务中能够准确地传递人脸动作。
Face manipulation techniques improve fast with the development of powerful image generation models. Two particular face manipulation methods, namely face swap and expression reenactment attract much attention for their flexibility and ease to generate high quality synthesis results. Recently, these two subjects are actively studied. However, most existing methods treat the two tasks separately, ignoring their underlying similarity. In this paper, we propose to tackle the two problems within a unified framework that achieves high quality synthesis results. The enabling component for our unified framework is the clean disentanglement of 3D pose, shape, and expression factors and then recombining them for different tasks accordingly. We then use the same set of 2D representations for face swap and expression reenactment tasks that are input to a common image translation model to directly generate the final synthetic images. Once trained, the proposed model can accomplish both face swap and expression reenactment tasks for previously unseen subjects. Comprehensive experiments and comparisons show that the proposed method achieves high fidelity results in multiple aspects, and it is especially good at faithfully preserving source facial shape in the face swap task, and accurately transferring facial movements in the expression reenactment task.