Bio-mimetic high-speed target localization with fused frame and event vision for edge application.
Bio-mimetic high-speed target localization with fused frame and event vision for edge application.
复制标题
DOI：
10.3389/fnins.2022.1010302
复制
发表时间：
2022
影响因子：
4.3
通讯作者：
Raychowdhury, Arijit
中科院分区：
文献类型：
--
作者：
Lele, Ashwin Sanjay;Fang, Yan;Anwar, Aqeel;Raychowdhury, Arijit
关键词：
自然界的进化已经磨练了捕食技能，需要定位和拦截快速移动的猎物。当前这一代机器人系统使用深度学习来模仿这些生物系统。在这种受限的空中边缘机器人上使用卷积神经网络（CNN）（帧流水线）对相机帧进行高速处理会受到资源限制。增加更多的计算资源也最终限制了相机帧速率下的吞吐量，因为仅帧的传统系统无法捕获环境的详细时间动态。生物启发的事件相机和尖峰神经网络（SNN）提供了一个异步传感器-处理器对（事件流水线），捕获场景的连续时间细节，以实现高速但准确性滞后。在这项工作中，我们提出了一个目标定位系统相结合的事件相机和SNN为基础的高速目标估计和基于帧的相机和CNN驱动的可靠的目标检测融合事件和帧管道的互补时空实力。我们的主要贡献之一是设计了一个SNN滤波器，该滤波器借鉴了家蝇自我运动消除的神经机制。它将前庭传感器与视觉融合，以消除与捕食者的自我运动相对应的活动。我们还将神经启发的多管道处理与灵长类动物和昆虫中的任务优化的多神经元通路结构相结合。该系统被验证优于CNN的处理使用捕食者无人机模拟在现实的3D虚拟环境。然后，该系统在现实世界的多无人机设置与仿真事件数据。随后，我们使用多摄像头和惯性测量单元（IMU）组件记录的实际传感数据来显示所需的工作，同时容忍视觉和IMU传感器中的现实噪声。我们分析了设计空间，以确定尖峰神经元，CNN模型的最佳参数，并检查它们对融合系统性能指标的影响。最后，我们将吞吐量控制SNN和融合网络映射到边缘兼容的Zynq-7000 FPGA上，以显示即使在有限的资源可用性下，每秒也有可能输出264个。这项工作可能会开辟新的研究方向，通过耦合神经科学发现启发的多种传感和处理方式，打破基于帧的计算机视觉的基本权衡。
Evolution has honed predatory skills in the natural world where localizing and intercepting fast-moving prey is required. The current generation of robotic systems mimics these biological systems using deep learning. High-speed processing of the camera frames using convolutional neural networks (CNN) (frame pipeline) on such constrained aerial edge-robots gets resource-limited. Adding more compute resources also eventually limits the throughput at the frame rate of the camera as frame-only traditional systems fail to capture the detailed temporal dynamics of the environment. Bio-inspired event cameras and spiking neural networks (SNN) provide an asynchronous sensor-processor pair (event pipeline) capturing the continuous temporal details of the scene for high-speed but lag in terms of accuracy. In this work, we propose a target localization system combining event-camera and SNN-based high-speed target estimation and frame-based camera and CNN-driven reliable object detection by fusing complementary spatio-temporal prowess of event and frame pipelines. One of our main contributions involves the design of an SNN filter that borrows from the neural mechanism for ego-motion cancelation in houseflies. It fuses the vestibular sensors with the vision to cancel the activity corresponding to the predator's self-motion. We also integrate the neuro-inspired multi-pipeline processing with task-optimized multi-neuronal pathway structure in primates and insects. The system is validated to outperform CNN-only processing using prey-predator drone simulations in realistic 3D virtual environments. The system is then demonstrated in a real-world multi-drone set-up with emulated event data. Subsequently, we use recorded actual sensory data from multi-camera and inertial measurement unit (IMU) assembly to show desired working while tolerating the realistic noise in vision and IMU sensors. We analyze the design space to identify optimal parameters for spiking neurons, CNN models, and for checking their effect on the performance metrics of the fused system. Finally, we map the throughput controlling SNN and fusion network on edge-compatible Zynq-7000 FPGA to show a potential 264 outputs per second even at constrained resource availability. This work may open new research directions by coupling multiple sensing and processing modalities inspired by discoveries in neuroscience to break fundamental trade-offs in frame-based computer vision1.
登录
查看更多内容
影响因子：
5.2
作者：
Gallego, Guillermo;Scaramuzza, Davide
通讯作者：
Scaramuzza, Davide
影响因子：
7.7
作者：
Förster D;Helmbrecht TO;Mearns DS;Jordan L;Mokayes N;Baier H
通讯作者：
Baier H
影响因子：
3.2
作者：
Diehl PU;Cook M
通讯作者：
Cook M
影响因子：
16.2
作者：
Gollisch, Tim;Meister, Markus
通讯作者：
Meister, Markus
DOI：
10.1109/tcad.2015.2474396
发表时间：
2015-10-01
影响因子：
2.9
作者：
Akopyan, Filipp;Sawada, Jun;Modha, Dharmendra S.
通讯作者：
Modha, Dharmendra S.