Implicit Neural Representations for Facial Animation

神经辐射场（nerf）是一种隐式神经表示，它使用神经网络对复杂的视觉数据建模，而无需明确定义其参数。近年来，它们已成为3D图形和资产生成的领先解决方案。它们的日益突出超越了静态3D建模，并且越来越多地被动画和虚拟现实（VR）社区所接受，主要是由于它们能够平衡高压缩性和卓越的重建精度。nerf在各种应用中都有很大的前景，其中一个引人入胜的领域是面部动画领域。尽管有其优点，但nerf也带来了一系列挑战，我们的目标是实现以下研究目标：可解释性和可控性的改进：nerf作为“黑匣子”模型运行，其不透明性使得难以理解和控制输出的特定物理属性。在面部动画的背景下，这可能导致对动画角色所传达的外观和情感的次优控制。为此，我们将开发面部身份和表情的解纠缠表示，以提高可控性。2. 语言整合：实现视觉和听觉线索之间的多模态同步是一项具有挑战性和持续的努力。我们将开发的模型不仅能捕捉面部表情的视觉动态，还能使它们与语音元素同步。3. 水印：nerf和类似的生成模型引起了人们对其潜在滥用的担忧，特别是在创建“深度伪造”内容时。深度造假可以用来操纵图像和视频，使其具有高度的真实感，从而对隐私、安全和错误信息构成风险。作为回应，我们的目标是开发健壮的水印和认证技术，这些技术对于检测和减轻“深度伪造”内容至关重要。动画人物形象有着广泛的应用——丰富的虚拟现实体验、电话会议、游戏和在线教育。基于nerf的面部动画的整合将提升现实主义水平，从而产生身临其境的VR环境。此外，数据驱动的方法将通过消除美工花费数小时手动设计角色的需要来简化工作流程。这不仅可以提高数字体验的质量，还可以加快创意过程，使nerf成为艺术家、开发者和用户的宝贵资产。我们在开发面部动画的神经表征方面的研究与EPSRC对“图像和视觉计算”的兴趣直接一致。近年来，生成模型已经成为工业界和学术界的技术“圣杯”，EPSRC已经在全国范围内资助了各种关于这一主题的研究项目，依靠人工智能模型的巨大潜力。通过推进面部动画的最新技术，我们的研究将扩展3D视觉和图形的边界。这项研究的结果有望增强视觉真实感和多模态交互性，对未来的人机交互和数字通信具有广泛的影响，充分反映了EPSRC在该领域推进创新的承诺。

Neural Radiance Fields (NeRFs) are a type of implicit neural representation that uses neural networks to model complex visual data without explicitly defining their parameters. In recent years, they have emerged as a leading solution in 3D graphics and asset generation. Their growing prominence extends beyond static 3D modelling and is increasingly embraced by the animation and virtual reality (VR) communities, primarily due to their ability to balance high compressibility and exceptional reconstruction accuracy. NeRFs hold substantial promise in a variety of applications, and one fascinating domain is the field of Facial Animation.Despite their merits, NeRFs come with their set of challenges and we aim to achieve the following research objectives: 1. Improvement in Explainability and Controllability: NeRFs operate as "black box" models and their opacity makes it difficult to understand and control the specific physical attributes of the output. In the context of facial animation, this could lead to suboptimal control over the appearance and emotions conveyed by animated avatars. To this end, we will develop disentangled representations of facial identity and expressions to improve controllability. 2. Integration of Speech: Achieving multimodal synchronization between visual and auditory cues is a challenging and ongoing endeavour. We will develop models that will not only capture the visual dynamics of facial expressions but also synchronize them with the phonetic elements of speech. 3. Watermarking: NeRFs and similar generative models have raised concerns about their potential misuse, especially in the creation of "deepfake" content. Deepfakes can be used to manipulate images and videos to a high degree of realism, posing risks to privacy, security, and misinformation. As a response, we aim to develop robust watermarking and authentication techniques that are crucial to detect and mitigate "deepfake" content. Animated human avatars have a wide array of applications - rich VR experiences, teleconferencing, gaming, and online education. The integration of NeRF-based facial animation would elevate the level of realism, resulting in an immersive VR environment. Moreover, the data-driven approach would streamline workflows by eliminating the need for artists to spend hours on manual avatar design. This would not only enhance the quality of digital experiences but also accelerate creative processes, making NeRFs an invaluable asset for artists, developers, and users alike. Our research on developing neural representations for facial animation directly aligns with EPSRC's interest in "Image and Vision Computing." In recent years, generative models have become the technological "holy grail" for industry and academia alike and EPSRC has been funding various research projects under this theme across the country, counting on the massive potential of AI models. By advancing the state-of-the-art in NeRFs for facial animation, our research would expand the boundaries of 3D vision and graphics. The outcomes of this research promise enhanced visual realism and multimodal interactivity, with broad implications for the future of human-computer interaction and digital communication, fully reflecting EPSRC's commitment to advancing innovations in the field.