Production-Ready Face Re-Aging for Visual Effects
Production-Ready Face Re-Aging for Visual Effects
复制标题
DOI：
10.1145/3550454.3555520
复制
发表时间：
2022-11
期刊：
影响因子：
--
通讯作者：
G. Zoss;Prashanth Chandran;Eftychios Sifakis;Markus H. Gross;Paulo F. U. Gotardo;D. Bradley
中科院分区：
文献类型：
--
作者：
G. Zoss;Prashanth Chandran;Eftychios Sifakis;Markus H. Gross;Paulo F. U. Gotardo;D. Bradley
在娱乐和广告中，对视频中的面部进行逼真的数字再老化变得越来越普遍。但是，主流的2D绘画工作流程通常需要一帧一帧的手动工作，即使是熟练的艺术家也需要几天才能完成。虽然对面部图像再老化的研究已经尝试自动化并解决这个问题，但是当前的技术几乎没有实际用途，因为它们通常遭受面部身份丢失、分辨率差以及后续视频帧中的不稳定结果。在本文中，我们提出了第一个实用的，全自动和生产准备的方法，重新老化的视频图像中的面孔。我们的第一个关键见解是解决收集纵向训练数据的问题，以学习在很长一段时间内重新老化面孔，这对于大量真实的人来说几乎是不可能完成的任务。我们展示了如何利用当前最先进的面部再老化技术构建这样一个纵向数据集，尽管在真实的图像上失败，但在合成面部上提供了照片真实的再老化结果。然后，我们的第二个关键见解是利用这些合成数据，并将面部重新老化作为一个实际的图像到图像的翻译任务，可以通过训练一个很好理解的U-Net架构来执行，而不需要更复杂的网络设计。我们演示了如何简单的U-网，令人惊讶的是，使我们能够推进最先进的重新老化视频上的真实的面孔，具有前所未有的时间稳定性和保存的面部身份在变量的表情，观点和照明条件。最后，我们的新面部再老化网络（FRAN）采用了简单直观的机制，为艺术家提供了本地化的控制和创作自由，以指导和微调再老化效果，这一功能在真实的生产流水线中非常重要，但在相关研究工作中经常被忽视。
Photorealistic digital re-aging of faces in video is becoming increasingly common in entertainment and advertising. But the predominant 2D painting workflow often requires frame-by-frame manual work that can take days to accomplish, even by skilled artists. Although research on facial image re-aging has attempted to automate and solve this problem, current techniques are of little practical use as they typically suffer from facial identity loss, poor resolution, and unstable results across subsequent video frames. In this paper, we present the first practical, fully-automatic and production-ready method for re-aging faces in video images. Our first key insight is in addressing the problem of collecting longitudinal training data for learning to re-age faces over extended periods of time, a task that is nearly impossible to accomplish for a large number of real people. We show how such a longitudinal dataset can be constructed by leveraging the current state-of-the-art in facial re-aging that, although failing on real images, does provide photoreal re-aging results on synthetic faces. Our second key insight is then to leverage such synthetic data and formulate facial re-aging as a practical image-to-image translation task that can be performed by training a well-understood U-Net architecture, without the need for more complex network designs. We demonstrate how the simple U-Net, surprisingly, allows us to advance the state of the art for re-aging real faces on video, with unprecedented temporal stability and preservation of facial identity across variable expressions, viewpoints, and lighting conditions. Finally, our new face re-aging network (FRAN) incorporates simple and intuitive mechanisms that provides artists with localized control and creative freedom to direct and fine-tune the re-aging effect, a feature that is largely important in real production pipelines and often overlooked in related research work.