Learning to See Through with Events
Learning to See Through with Events
复制标题
学会看透事件
DOI：
10.1109/tpami.2022.3227448
复制
发表时间：
2022
影响因子：
23.6
通讯作者：
Gui-Song Xia
中科院分区：
文献类型：
--
作者：
Lei Yu;Xiang Zhang;Wei Liao;Wen Yang;Gui-Song Xia
虽然合成孔径成像（SAI）可以通过模糊掉离焦的前景遮挡而实现透视效果，同时从多视点图像中恢复聚焦的遮挡场景，但是其性能通常会受到密集遮挡和极端光照条件的影响。为了解决这个问题，本文提出了一种基于事件的SAI（E-SAI）方法，依靠异步事件具有极低的延迟和高动态范围的事件摄像机获得。具体来说，收集的事件首先由aRefocus-Netmodule重新聚焦，以对齐焦点内事件，同时分散散焦事件。在此基础上，提出了一种由脉冲神经网络（SNNs）和卷积神经网络（CNN）组成的混合网络，用于对来自重聚焦事件的时空信息进行编码，并重建被遮挡目标的视觉图像。大量的实验表明，我们提出的E-SAI方法可以在处理非常密集的遮挡和极端光照条件下取得显着的性能，并从纯事件产生高质量的图像。代码和数据集可在https://dvs-whu.cn/projects/esai/上获得。
Although synthetic aperture imaging (SAI) can achieve the seeing-through effect by blurring out off-focus foreground occlusions while recovering in-focus occluded scenes from multi-view images, its performance is often deteriorated by dense occlusions and extreme lighting conditions. To address the problem, this paper presents an Event-based SAI (E-SAI) method by relying on the asynchronous events with extremely low latency and high dynamic range acquired by an event camera. Specifically, the collected events are first refocused by aRefocus-Netmodule to align in-focus events while scattering out off-focus ones. Following that, ahybrid networkcomposed of spiking neural networks (SNNs) and convolutional neural networks (CNNs) is proposed to encode the spatio-temporal information from the refocused events and reconstruct a visual image of the occluded targets. Extensive experiments demonstrate that our proposed E-SAI method can achieve remarkable performance in dealing with very dense occlusions and extreme lighting conditions and produce high-quality images from pure events. Codes and datasets are available at https://dvs-whu.cn/projects/esai/ .