Progressive Temporal Feature Alignment Network for Video Inpainting
Progressive Temporal Feature Alignment Network for Video Inpainting
复制标题
DOI：
10.1109/cvpr46437.2021.01618
复制
发表时间：
2021-04
期刊：
影响因子：
--
通讯作者：
Xueyan Zou;Linjie Yang;Ding Liu;Yong Jae Lee
中科院分区：
文献类型：
--
作者：
Xueyan Zou;Linjie Yang;Ding Liu;Yong Jae Lee
视频修复旨在用可信的内容填充时空“损坏”区域。为了实现这一目标，有必要从相邻帧中找到对应关系，以忠实地产生未知内容。目前的方法通过注意力、基于流的扭曲或3D时间卷积来实现这一目标。然而，当光流不准确时，基于流的扭曲可能产生伪影，而时间卷积可能遭受空间未对准。我们提出了“渐进的时间特征对齐网络”，它逐步丰富的功能，从当前帧提取的特征与特征扭曲从相邻帧使用光流。我们的方法纠正了时间特征传播阶段的空间错位，大大提高了视觉质量和时间一致性的修复视频。与现有的深度学习方法相比，使用所提出的架构，我们在DAVIS和FVI数据集上实现了最先进的性能。代码可在https://github.com/MaureenZOU/TSAM上获得。
Video inpainting aims to fill spatiotemporal "corrupted" regions with plausible content. To achieve this goal, it is necessary to find correspondences from neighbouring frames to faithfully hallucinate the unknown con-tent. Current methods achieve this goal through attention, flow-based warping, or 3D temporal convolution. However, flow-based warping can create artifacts when optical flow is not accurate, while temporal convolution may suffer from spatial misalignment. We propose ‘Progressive Temporal Feature Alignment Network’, which progressively enriches features extracted from the current frame with the feature warped from neighbouring frames using optical flow. Our approach corrects the spatial misalignment in the temporal feature propagation stage, greatly improving visual quality and temporal consistency of the inpainted videos. Using the proposed architecture, we achieve state-of-the-art performance on the DAVIS and FVI datasets compared to existing deep learning approaches. Code is available at https://github.com/MaureenZOU/TSAM.