Controllable Radiance Fields for Dynamic Face Synthesis
Controllable Radiance Fields for Dynamic Face Synthesis
复制标题
DOI：
10.1109/3dv57658.2022.00075
复制
发表时间：
2022-09
期刊：
影响因子：
--
通讯作者：
Peiye Zhuang;Liqian Ma;Oluwasanmi Koyejo;A. Schwing
中科院分区：
文献类型：
--
作者：
Peiye Zhuang;Liqian Ma;Oluwasanmi Koyejo;A. Schwing
最近在3D感知图像合成方面的工作已经取得了令人信服的结果，使用神经渲染的进步。然而，三维感知的人脸动态合成还没有得到太多的关注。在这里，我们研究如何显式地控制表现出非刚性运动（例如，面部表情变化），同时确保3D感知。为此，我们提出了一个可控辐射场（CoRF）：1）运动控制是通过嵌入运动特征内的分层潜在的运动空间的风格为基础的发电机; 2）为了确保一致性的背景，运动特征和主题特定的属性，如照明，纹理，形状，轮廓，和身份，面部解析网络，头部回归器和身份编码器。在头部图像/视频数据上，我们表明CoRFs是3D感知的，同时能够编辑身份，观看方向和运动。
Recent work on 3D-aware image synthesis has achieved compelling results using advances in neural rendering. However, 3D-aware synthesis of face dynamics hasn't received much attention. Here, we study how to explicitly control generative model synthesis of face dynamics exhibiting non-rigid motion (e.g., facial expression change), while simultaneously ensuring 3D-awareness. For this we propose a Controllable Radiance Field (CoRF): 1) Motion control is achieved by embedding motion features within the layered latent motion space of a style-based generator; 2) To ensure consistency of background, motion features and subject-specific attributes such as lighting, texture, shapes, albedo, and identity, a face parsing net, a head regressor and an identity encoder are incorporated. On head image/video data we show that CoRFs are 3D-aware while enabling editing of identity, viewing directions, and motion.