End-to-End Learning of Video Super-Resolution with Motion Compensation
End-to-End Learning of Video Super-Resolution with Motion Compensation
复制标题
DOI：
10.1007/978-3-319-66709-6_17
复制
发表时间：
2017-07
期刊：
影响因子：
--
通讯作者：
Osama Makansi;Eddy Ilg;T. Brox
中科院分区：
文献类型：
--
作者：
Osama Makansi;Eddy Ilg;T. Brox
学习方法已经在给定低分辨率输入的超分辨率图像的任务中显示出巨大的成功。视频超分辨率的目的是利用额外的信息，从多个图像。通常，图像通过光流和连续图像扭曲相关。在本文中，我们提供了一个端到端的视频超分辨率网络，与以前的作品相比，包括在整个网络架构的光流估计。我们分析了光流用于视频超分辨率的使用，发现常见的现成的图像扭曲不允许视频超分辨率从光流中受益。相反，我们提出了一种用于运动补偿的操作，其直接执行从低到高分辨率的扭曲。我们表明，这种网络配置，视频超分辨率可以受益于光流，我们获得了最先进的结果，流行的测试集。我们还表明，整个图像的处理，而不是独立的补丁是负责一个大的准确性增加。
Learning approaches have shown great success in the task of super-resolving an image given a low resolution input. Video super-resolution aims for exploiting additionally the information from multiple images. Typically, the images are related via optical flow and consecutive image warping. In this paper, we provide an end-to-end video super-resolution network that, in contrast to previous works, includes the estimation of optical flow in the overall network architecture. We analyze the usage of optical flow for video super-resolution and find that common off-the-shelf image warping does not allow video super-resolution to benefit much from optical flow. We rather propose an operation for motion compensation that performs warping from low to high resolution directly. We show that with this network configuration, video super-resolution can benefit from optical flow and we obtain state-of-the-art results on the popular test sets. We also show that the processing of whole images rather than independent patches is responsible for a large increase in accuracy.