Video Rewrite: Driving Visual Speech with Audio
Video Rewrite: Driving Visual Speech with Audio
复制标题
DOI：
10.1145/3596711.3596787
复制
发表时间：
1997-08
期刊：
影响因子：
--
通讯作者：
C. Bregler;M. Covell;M. Slaney
中科院分区：
文献类型：
--
作者：
C. Bregler;M. Covell;M. Slaney
视频重写使用现有的镜头自动创建一个新的视频，其中一个人说出她在原始镜头中没有说过的话。该技术在电影配音中很有用，例如，可以修改电影序列以使演员的嘴唇动作与新的配乐同步。视频重写会自动标记训练数据和新音轨中的音素。视频重写对训练片段中的嘴部图像进行重新排序，以匹配新音轨的音素序列。当训练片段中没有特定音素时，视频重写会选择最接近的近似值。由此产生的口腔图像序列被拼接到背景镜头中。该拼接过程会自动纠正嘴部图像和背景镜头之间头部位置和方向的差异。视频重写使用计算机视觉技术来跟踪训练片段中说话者嘴部的点，并使用变形技术将这些嘴部姿势组合到最终的视频序列中。新视频将原演员的动态发音与背景镜头所规定的举止和场景结合在一起。 Video Rewrite 是第一个面部动画系统，可自动执行将现有素材重新同步到新配乐所需的所有标签和组装任务。
Video Rewrite uses existing footage to create automatically new video of a person mouthing words that she did not speak in the original footage. This technique is useful in movie dubbing, for example, where the movie sequence can be modified to sync the actors' lip motions to the new soundtrack. Video Rewrite automatically labels the phonemes in the training data and in the new audio track. Video Rewrite reorders the mouth images in the training footage to match the phoneme sequence of the new audio track. When particular phonemes are unavailable in the training footage, Video Rewrite selects the closest approximations. The resulting sequence of mouth images is stitched into the background footage. This stitching process automatically corrects for differences in head position and orientation between the mouth images and the background footage. Video Rewrite uses computer-vision techniques to track points on the speaker's mouth in the training footage, and morphing techniques to combine these mouth gestures into the final video sequence. The new video combines the dynamics of the original actor's articulations with the mannerisms and setting dictated by the background footage. Video Rewrite is the first facial-animation system to automate all the labeling and assembly tasks required to resync existing footage to a new soundtrack.