Event-guided Video Clip Generation from Blurry Images
Event-guided Video Clip Generation from Blurry Images
复制标题
DOI：
10.1145/3503161.3548142
复制
发表时间：
2022-10
期刊：
影响因子：
--
通讯作者：
Xin Ding;T. Takatani;Zhongyuan Wang;Ying Fu;Yinqiang Zheng
中科院分区：
文献类型：
--
作者：
Xin Ding;T. Takatani;Zhongyuan Wang;Ying Fu;Yinqiang Zheng
动态和有源像素视觉传感器（DAVIS）可以同时产生由动态视觉传感器（DVS）捕获的异步事件流和来自有源像素传感器（APS）的强度帧。事件序列具有高的时间分辨率和高的动态范围，而强度图像容易遭受运动模糊，由于APS的帧速率低。在本文中，我们提出了一种基于端到端卷积神经网络的方法，该方法在事件的局部和全局约束下，通过从模糊图像及其相关事件流中进行协作学习来恢复清晰，清晰的强度帧。具体来说，我们首先学习清晰强度帧与具有其事件数据的对应模糊图像之间的关系的函数。然后，我们提出了一个生成模块来实现它与监督模块，以约束在运动过程中的恢复。我们还通过同步DAVIS相机和高速相机来捕获具有成对模糊帧/事件和锐利帧的第一个真实数据集。实验结果表明，我们的方法可以重建高质量的清晰的视频片段，并优于国家的最先进的模拟和真实世界的数据。
Dynamic and active pixel vision sensors (DAVIS) can simultaneously produce streams of asynchronous events captured by the dynamic vision sensor (DVS) and intensity frames from the active pixel sensor (APS). Event sequences show high temporal resolution and high dynamic range, while intensity images easily suffer from motion blur due to the low frame rate of APS. In this paper, we present an end-to-end convolutional neural network based method under the local and global constraints of events to restore clear, sharp intensity frames through collaborative learning from a blurry image and its associated event streams. Specifically, we first learn a function of the relationship between the sharp intensity frame and the corresponding blurry image with its event data. Then we propose a generation module to realize it with a supervision module to constrain the restoration in the motion process. We also capture the first realistic dataset with paired blurry frame/events and sharp frames by synchronizing a DAVIS camera and a high-speed camera. Experimental results show that our method can reconstruct high-quality sharp video clips, and outperform the state-of-the-art on both simulated and real-world data.