Unsupervised Learning of Dense Optical Flow, Depth and Egomotion with Event-Based Sensors
Unsupervised Learning of Dense Optical Flow, Depth and Egomotion with Event-Based Sensors
复制标题
使用基于事件的传感器对密集光流、深度和自我运动进行无监督学习
DOI：
--
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
Y. Aloimonos
中科院分区：
文献类型：
--
作者：
Chengxi Ye;A. Mitrokhin;C. Fermüller;J. Yorke;Y. Aloimonos
我们提出了一个无监督的学习管道，用于自动驾驶应用的密集深度，光流和自运动估计，使用动态视觉传感器（DVS）的基于事件的输出作为输入。我们的管道的骨干是一个生物启发的编码器-解码器神经网络架构- ECN。为了训练管道，我们引入了一种协方差归一化技术，类似于动物神经系统中的侧抑制机制。我们的工作是第一个单目管道，仅从稀疏事件数据生成密集的深度和光流，并且能够从白天转移到夜晚场景，而无需任何额外的训练。该网络以自我监督模式工作，只有15万个参数。我们在MVSEC自动驾驶数据集上评估了我们的管道，并给出了深度，光流和自运动估计的结果。得益于高效的设计，我们能够在单个Nvidia 1080Ti GPU上实现300 FPS的推理速率。我们的实验表明，在对事件数据进行深度学习的基础上，工作有了显著的改进，并且能够在白天和晚上都表现良好。
We present an unsupervised learning pipeline for dense depth, optical flow and egomotion estimation for autonomous driving applications, using the event-based output of the Dynamic Vision Sensor (DVS) as input. The backbone of our pipeline is a bioinspired encoder-decoder neural network architecture - ECN. To train the pipeline, we introduce a covariance normalization technique which resembles the lateral inhibition mechanism found in animal neural systems.Our work is the first monocular pipeline that generates dense depth and optical flow from sparse event data only, and is able to transfer from day to night scenes without any additional training. The network works in self-supervised mode and has just 150k parameters. We evaluate our pipeline on the MVSEC self driving dataset and present results for depth, optical flow and and egomotion estimation. Thanks to the efficient design, we are able to achieve inference rates of 300 FPS on a single Nvidia 1080Ti GPU. Our experiments demonstrate significant improvements upon works that used deep learning on event data, as well as the ability to perform well during both day and night.