High-fidelity, directable animation transfer using facial decomposition on optimized micro-sequences

娱乐业向说大量不同语言的人提供世界各地的媒体产品。媒体产品，如电视或互联网广告、电影和视频游戏，通常由视频和音频流组成。为了适应不同的语言，制作人通常会在后期制作步骤中以不同的语言重新生成声音流，这一过程被称为配音。虽然重新生成音频流(配音)仍然需要大量的努力，但从研究和生产的角度来看，仍然存在的一个悬而未决的挑战是如何将原始视频流与新的音频流同步。由于演员的嘴唇动作与配乐不一致，缺乏同步产生了一种非常诡异的效果，使人失去了体验。在这个项目中，我们为配音同步问题提出了一个实用的解决方案，产生了可以在制片厂使用的自然结果。为了实现这一目标，我们与位于蒙特利尔的制作工作室AudioZ建立了战略合作伙伴关系，我们将在这个项目上与AudioZ合作。我们提出了一种视频合成方法，该方法融合了聚焦于嘴唇区域的动画，同时保留了原始演员的面部表情。我们的方法使用了新的优化技术来避免视频合成中常见的不受欢迎的诡异山谷效应。

The entertainment industry delivers media products around the world to people speaking a large number of distinct languages. Media products such as TV or internet spots, movies and video games, typically consist of a video and sound stream. To accommodate different languages, the producers typically re-generate the sound stream in different language in a post-production step, a process known as dubbing. While it still takes a lot of effort to re-generate the sound stream (dubbing), one of the remaining open challenges both from a research and production point of view is how to synchronize the original video stream with the new audio stream. The lack of synchronization creates a very uncanny effect that takes away from the experience since the lip motion of the actors does not correspond to the soundtrack. In this project we propose a practical solution to the dubbing synchronization problem, producing natural results that can be used in a production studio. To achieve that, we created a strategic partnership with AudioZ, a production studio located in Montreal with whom we will collaborate on this project. We propose a video synthesis method that blends animation focusing on the lip area while preserving the facial expression of the original actor. Our method uses novel optimization techniques to avoid the common undesired uncanny valley effect of video synthesis.