EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras
EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras
复制标题
DOI：
10.15607/rss.2018.xiv.062
复制
发表时间：
2018-02
期刊：
影响因子：
--
通讯作者：
A. Z. Zhu;Liangzhe Yuan;Kenneth Chaney;Kostas Daniilidis
中科院分区：
文献类型：
--
作者：
A. Z. Zhu;Liangzhe Yuan;Kenneth Chaney;Kostas Daniilidis
基于事件的相机在基于帧的相机受到影响的各种情况下（诸如高速运动和高动态范围场景）已经显示出很大的前景。然而，开发用于事件测量的算法需要一类新的手工算法。深度学习在为视觉社区中的许多问题提供无模型解决方案方面取得了巨大成功，但现有的网络是在考虑基于帧的图像的情况下开发的，并且不存在用于监督训练的图像所具有的丰富的事件标记数据。针对这些问题，我们提出了EV-FlowNet，这是一种新型的自监督深度学习管道，用于基于事件的相机的光流估计。特别是，我们引入了一个给定的事件流，这是一个自监督神经网络作为唯一的输入馈入基于图像的表示。然后，在给定来自网络的估计流量的情况下，将与事件同时从同一相机捕获的对应灰度图像用作监督信号以在训练时间提供损失函数。我们表明，由此产生的网络能够准确预测仅在各种不同场景中事件的光流，其性能与基于图像的网络相竞争。该方法不仅可以准确估计密集光流，而且还提供了一个框架，其他自监督方法的转移到基于事件的域。
Event-based cameras have shown great promise in a variety of situations where frame based cameras suffer, such as high speed motions and high dynamic range scenes. However, developing algorithms for event measurements requires a new class of hand crafted algorithms. Deep learning has shown great success in providing model free solutions to many problems in the vision community, but existing networks have been developed with frame based images in mind, and there does not exist the wealth of labeled data for events as there does for images for supervised training. To these points, we present EV-FlowNet, a novel self-supervised deep learning pipeline for optical flow estimation for event based cameras. In particular, we introduce an image based representation of a given event stream, which is fed into a self-supervised neural network as the sole input. The corresponding grayscale images captured from the same camera at the same time as the events are then used as a supervisory signal to provide a loss function at training time, given the estimated flow from the network. We show that the resulting network is able to accurately predict optical flow from events only in a variety of different scenes, with performance competitive to image based networks. This method not only allows for accurate estimation of dense optical flow, but also provides a framework for the transfer of other self-supervised methods to the event-based domain.