CRII: FET: Neuromorphic Processing Framework for Spatiotemporal Fusion of Visual Sensors

CRII：FET：视觉传感器时空融合的神经形态处理框架

• 批准号: 2153440
• 负责人: Yan Fang
• 金额: $ 17.39万
• 依托单位: Kennesaw State University Research and Service Foundation
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-04-01 至 2025-03-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2153440&HistoricalAwards=false
• 关键词: CRII; FET; Neuromorphic; Processing; Framework

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Visual processing tasks such as detection, tracking, and localization are essential to the automation of unmanned aerial vehicles (UAV), robots, surveillance, and defense systems. However, these intelligent tasks become challenging on high-speed motion and edge devices due to limited computing resources and low power supplies. This research will explore a brain-inspired framework to process the visual information from two complementary visual sensors, event-based dynamical vision sensors (DVS) and frame-based standard cameras, in a sensor-fusion style. The overarching goal is to address the challenge of high-speed and energy-efficient visual processing with end-to-end closed-loop control on edge computing systems. The proposed research will benefit numerous robotics, surveillance, IoT security, and national defense applications. This work will also explore novel hybrid neural networks, thus contributing to the quest to general AI and enhancing the interdisciplinary collaboration between computer science and neuroscience. To encourage young students in this research, the project will 1) design hands-on projects and course modules related to DVS cameras and neuromorphic algorithms, 2) initiate K-12 education outreach for local minority high school students through ongoing University Programs, and 3) recruit minority undergraduate researchers. The proposed project will exploit the synergy of two brain-inspired learning models, neuromorphic spiking neural networks and regular deep neural networks. Such a hybrid neuromorphic framework can harness the high spatial resolution from a standard camera and the high temporal resolution from a DVS camera. The temporal encoded data from the DVS camera is suitable to be processed in a spiking neural network. In contrast, the data from the standard camera are compatible with traditional convolutional networks. This proposal will 1) design a hybrid neuromorphic framework composed of spiking neural networks and conventional artificial neural networks to process event-frame fused visual data; 2) adapt such a framework to UAV or robots and develop an end-to-end close-loop neuromorphic platform for various high-speed visual tasks; and 3) explore the model compression of hybrid neural networks and architecture design of the hardware accelerator for the proposed framework.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该奖项全部或部分由2021年美国救援计划法案（公法117-2）资助。检测、跟踪和定位等视觉处理任务对于无人机（UAV）、机器人、监视和防御系统的自动化至关重要。 然而，由于有限的计算资源和低电源，这些智能任务在高速运动和边缘设备上变得具有挑战性。本研究将探索一种大脑启发的框架，以传感器融合的方式处理来自两个互补视觉传感器（基于事件的动态视觉传感器（DVS）和基于帧的标准相机）的视觉信息。总体目标是通过边缘计算系统的端到端闭环控制来解决高速和节能视觉处理的挑战。拟议的研究将有利于许多机器人，监控，物联网安全和国防应用。这项工作还将探索新的混合神经网络，从而有助于探索通用人工智能，并加强计算机科学和神经科学之间的跨学科合作。为了鼓励年轻学生参与这项研究，该项目将1）设计与DVS相机和神经形态算法相关的实践项目和课程模块，2）通过正在进行的大学课程为当地少数民族高中生开展K-12教育推广，3）招募少数民族本科研究人员。该项目将利用两种大脑启发学习模型的协同作用，神经形态尖峰神经网络和常规深度神经网络。这种混合神经形态框架可以利用来自标准相机的高空间分辨率和来自DVS相机的高时间分辨率。来自DVS相机的时间编码数据适合于在尖峰神经网络中处理。相比之下，来自标准相机的数据与传统的卷积网络兼容。本文的研究目标是：1）设计一个由脉冲神经网络和传统人工神经网络组成的混合神经形态框架，用于处理事件-框架融合的视觉数据; 2）将该框架应用于无人机或机器人，开发一个端到端的闭环神经形态平台，用于各种高速视觉任务;和3）探索混合神经网络的模型压缩和拟议框架的硬件加速器的架构设计。该奖项反映了NSF的法定使命，并已被视为通过使用基金会的知识价值和更广泛的影响审查标准进行评估，

项目成果

Solving Quadratic Unconstrained Binary Optimization with Collaborative Spiking Neural Networks

使用协作尖峰神经网络求解二次无约束二元优化

• DOI: 10.1109/icrc57508.2022.00021
• 发表时间: 2023
• 期刊: 2022 IEEE International Conference on Rebooting Computing (ICRC
• 作者: Fang, Yan;Lele, Ashwin Sanjay
• 通讯作者: Lele, Ashwin Sanjay

Live Demonstration: Hybrid RRAM and SRAM SoC for Fused Frame and Event Target Tracking

现场演示：用于融合帧和事件目标跟踪的混合 RRAM 和 SRAM SoC

• DOI: 10.1109/iscas46773.2023.10181482
• 发表时间: 2023
• 期刊: IEEE
• 作者: Lele, Ashwin;Chang, Muya;Spetalnick, Samuel;Fang, Yan;Crafton, Brian;Konno, Shota;Raychowdhury, Arijit
• 通讯作者: Raychowdhury, Arijit

Memory-Based Computing for Energy-Efficient AI: Grand Challenges

• DOI: 10.1109/vlsi-soc57769.2023.10321880
• 发表时间: 2023-10
• 期刊: 2023 IFIP/IEEE 31st International Conference on Very Large Scale Integration (VLSI-SoC)
• 作者: Foroozan Karimzadeh;Mohsen Imani;Bahar Asgari;Ningyuan Cao;Yingyan Lin;Yan Fang

Incorporating Event Information for High-Quality Video Interpolation

• DOI: 10.1109/icnsc58704.2023.10319006
• 发表时间: 2023-10
• 期刊: 2023 IEEE International Conference on Networking, Sensing and Control (ICNSC)
• 作者: Beibei Yang;Guangyu Jiang;Yan Fang;Weiling Li

Neuromorphic Swarm on RRAM Compute-in-Memory Processor for Solving QUBO Problem

RRAM 内存计算处理器上的神经形态群用于解决 QUBO 问题

• DOI: 10.1109/dac56929.2023.10247852
• 发表时间: 2023
• 期刊: IEEE
• 作者: Lele, Ashwin Sanjay;Chang, Muya;Spetalnick, Samuel D.;Crafton, Brian;Raychowdhury, Arijit;Fang, Yan
• 通讯作者: Fang, Yan