Generative Adversarial Network for Text-to-Face Synthesis and Manipulation with Pretrained BERT Model
Generative Adversarial Network for Text-to-Face Synthesis and Manipulation with Pretrained BERT Model
复制标题
DOI：
10.1109/fg52635.2021.9666791
复制
发表时间：
2021-12
期刊：
影响因子：
--
通讯作者：
Yutong Zhou;N. Shimada
中科院分区：
文献类型：
--
作者：
Yutong Zhou;N. Shimada
这项工作提出了一种循环生成对抗网络，具有空间和通道注意力模块，用于文本到面部的合成和操作。然后，我们探索预训练的基于transformer的BERT模型来获得文本嵌入。此外，双层知觉损失和SSIM损失的引入，以加强微妙的功能，并保持在操作任务的面部身份。此外，我们采用了一种新的Flickr-Faces-HQ with Text descriptions（FFHQ-Text）数据集，其中包含许多面部属性注释，以推进文本到面部任务的开发。特别是，通过将用于学习潜在表示的StyleGAN编码器引入我们提出的后处理方法，我们证明了即使在较小的文本到人脸数据集上进行训练也可以合成更逼真的图像。实验结果表明，我们的方法，生成照片般逼真的面部图像，编辑与相关的关键字操作的特定面部属性的有效性，优于以前的国家的最先进的方法在质量和数量，并提出了有前途的未来方向。
This work proposes a cyclic generative adversarial network with spatial-wise and channel-wise attention modules for text-to-face synthesis and manipulation. Then, we explore the pre-trained transformer-based BERT model to obtain text embedding. Furthermore, dual-layer perceptual loss and SSIM loss are introduced to reinforce the delicate features and preserve facial identity during the manipulation task. Additionally, we adopt a novel Flickr-Faces-HQ with Text descriptions (FFHQ-Text) dataset with numerous facial attribute annotations to advance the development of the text-to-face task. In particular, by introducing the StyleGAN encoder for learning latent representations to our proposed post-processing method, we demonstrate that even training on a smaller text-to-face dataset can synthesize more realistic images. Experimental results reveal the effectiveness of our approach, which generates photo-realistic facial images, edits the specific facial attribute with the correlated keywords manipulation, outperforms previous state-of-the-art methods both in quality and quantity, and suggests promising future directions.