Visual understanding of faces in motion

对运动中面部的视觉理解

• 批准号: 133525
• 金额: $ 8.84万
• 依托单位: SYNTHESIA LIMITED
• 依托单位国家: 英国
• 项目类别: Feasibility Studies
• 财政年份: 2018
• 资助国家: 英国
• 起止时间: 2018 至 无数据
• 项目状态: 已结题
• 来源: https://gtr.ukri.org/projects?ref=133525
• 关键词: Visual; understanding; faces; motion

Recent days have seen an explosion of markerless facial performance capture methods that can reconstruct the dense geometry of the face from multi-view or even a single video. Unfortunately, even the best of these methods fail to capture the extensive range of shapes and deformations displayed by lips in motion, which results in a lack of expression in video synthesis of faces in speech. Lips exhibit agile and highly deformable motions which are extremely hard to capture -- at the same time, subtle lip motions are often the main vehicle for humans to convey emotions. **Capturing 3D lip motion with high accuracy** is therefore crucial if we are to design systems that can **synthesise realistic human speech** to convey human-like emotions.**The key goal and main innovation in VISIM will be to construct a low-cost, class-leading, three-dimensional (3D) morphable model (3DMM) of the face that focuses on representing the detailed dynamics of the mouth in speech to achieve unprecedented levels of realism in video synthesis of human speech.**We will achieve this by constructing a lightweight, multi-view, high frame-rate, video capture system to reconstruct the motion of the mouth and perform further lip shape refinement using photometric consistency and exploiting the redundancy in multiple views. We will use this to capture a range of subjects and train a new, richer 3DMM that encodes the detailed 3D dynamics of different people in speech.The **business value for Synthesia** is clear: this new, highly expressive 3DMM model will be instrumental in allowing us to **automate** the accurate capture of detailed 3D lip motion just from **monocular video** using an analysis-by-synthesis approach. We will then apply this to high-end facial video synthesis for media and entertainment industries. VISIM will create impact by bringing the high-levels of realism at the affordable price of a low-cost, video-based, automated capture system. This will pave the way to democratising high-end facial performance capture, allowing it to be adopted in new markets such as language dubbing for the TV, movie and advertising industries, where photorealistic results are critical but production budgets have so far restricted access to high-end content creation. This technology will enable Synthesia, a newly formed UK SME, to establish a new market in 'natural' language dubbing with photorealistic synthesis of the face.Keywords: Facial Performance Capture, Video-Based Tracking, Digital Content Creation.

最近几天，出现了无标记面部行为捕捉方法的爆炸性增长，这些方法可以从多个视图甚至单个视频重建密集的面部几何形状。不幸的是，即使是最好的方法也不能捕捉到嘴唇在运动中所表现出的广泛的形状和变形，这导致在语音中的视频合成中缺乏表情。嘴唇表现出极难捕捉的灵活和高度变形的动作--同时，微妙的嘴唇动作通常是人类传达情感的主要工具。因此，如果我们要设计能够**合成逼真的人类语音**以传达与人类相似的情感的系统，**高精度捕捉3D嘴唇运动**是至关重要的。**VISIM的关键目标和主要创新将是构建一个低成本、类领先的三维(3D)可变形人脸模型(3DMM)，该模型专注于表示语音中嘴部的详细动态，以在人类语音的视频合成中达到前所未有的逼真程度。**我们将通过构建一个轻量级、多视角、高帧速率、视频捕获系统，用于重建嘴巴的运动，并使用光度一致性和利用多个视图中的冗余来执行进一步的嘴唇形状细化。我们将使用它来捕获一系列主题，并训练一个新的、更丰富的3DMM，它编码不同人说话时的详细3D动态。**合成的商业价值**是明确的：这个新的、高度表达的3DMM模型将有助于我们使用合成分析方法**自动**从**单目视频**准确捕获详细的3D嘴唇动作。然后，我们将把它应用于媒体和娱乐行业的高端面部视频合成。VISIM将以负担得起的低成本、基于视频的自动化捕获系统的价格带来高水平的真实感，从而产生影响。这将为高端面部行为捕捉的民主化铺平道路，使其能够在电视、电影和广告行业的语言配音等新市场采用，这些行业的照片逼真效果至关重要，但制作预算迄今限制了高端内容创作的机会。这项技术将使Synthesia，一家新成立的英国中小企业，通过对面部进行照片级真实感合成，在自然语言配音方面建立一个新的市场。关键词：面部表演捕捉，基于视频的跟踪，数字内容创作。