CAREER: Reinventing Computer Vision through Bio-inspired Retinomorphic Vision Sensors, Corticomorphic Compute-In-Memory Processors and Event-based Algorithms

最先进的计算机视觉（CV）流水线是计算/内存密集型和功耗高的，这使得它们不适合高速应用，例如高超音速导弹跟踪或资源缺乏的边缘应用，例如由于尺寸，重量和功率（SWaP）限制的自主无人机导航。神经形态工程是一个很有前途的前沿，可以引入下一代CV系统，利用输入和网络架构中的稀疏性，通过基于事件的计算减少操作数量，即，只在必要时计算。该项目旨在通过开发一个紧密结合的系统，从具有丰富时空动态的设备和电路到受视觉皮层和视觉感知自适应学习算法启发的网络架构，开发一个多功能的节能生物感应，计算和学习框架。这将主要使用内存计算（CIM）架构来实现，该架构处理和提取各种关键的视觉特征，这些特征与存储在内存中的数据非常接近。拟议的研究将采用一种独特的集成方法，解决各个层面的挑战，从设备，电路，架构和算法到新颖的CV应用，灵感来自神经科学，例如低延迟动态对象分类，跟踪和自适应视觉注意力。有效培训计算机视觉神经形态工程新干部所需的技能组合的广度使得课程设计和与现有框架的整合具有极大的挑战性。拟议中的BioVision教育联盟将解决这一问题。该联盟的主要目标是合作并实施一项全面的劳动力发展计划，其中包括基于证据的最佳实践，以帮助培训新一代工程师和研究人员，以满足计算机视觉行业日益增长的需求。该提案的宏伟愿景是重新构想当今存在的现代计算机视觉（CV）管道，并使用生物启发传感器取代组件，处理器和算法，可以大大提高能源效率，数据效率和降低延迟。为了重塑CV管道，将同时解决三个研究重点。Thrust 1将专注于创建和构建一类新的视网膜视觉传感器，该传感器在功能，效率和延迟方面优于现有的相机，例如基于帧或神经形态的动态视觉传感器（DVS）。Thrust 2将专注于在硬件上对可扩展的皮质形态网络进行建模、设计和实现，使用混合反馈控制在多个时间尺度上展示非线性神经调节动力学。Thrust 3将专注于受神经科学启发的网络架构和算法的实现，例如具有随机奖励的强化学习，基于事件的时间模式识别。这项研究有可能在计算机视觉、神经形态计算和人工智能领域引发一代人的转变。开发一种节能的基于事件的相机，能够进行多功能时空模式识别和受视网膜启发的新功能，沿着通用的，可编程的，基于事件的计算机视觉管道，可以通过影响医疗保健，物联网（IoT），军事防御，边缘计算和工业自动化等关键领域，对我们的社会产生变革性的影响。在个人电子产品上使用先进的CV可以通过自动驾驶汽车、始终在线的智能监控和虚拟/增强现实（VR/AR）应用等技术彻底改变我们的生活方式。生物启发的视觉传感器，如Prophesee和iniVation等公司销售的DVS相机，主要在欧洲和亚洲开发，没有来自美国的工业或学术贡献。该计划将通过培养新一代世界级的研究人员来应对这一国家挑战，并为美国在部署下一代计算机视觉系统方面提供领先优势。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

State-of-the-art computer vision (CV) pipelines are compute/memory intensive and power hungry making them unsuitable for high-speed applications such as hypersonic missile tracking or resource-deficit edge applications such as autonomous drone navigation due to size, weight and power (SWaP) constraints. Neuromorphic engineering is a promising frontier to usher in the next generation of CV systems taking advantage of sparsity in the input and network architecture, reducing the number of operations through event-based computation i.e., compute only when necessary. This project aims to develop a versatile energy-efficient bio-inspired sensing, computing, and learning framework by developing a closely-knit system, from devices and circuits with rich spatio-temporal dynamics to network architectures inspired by the visual cortex and adaptive learning algorithms for visual perception. This will be achieved primarily using compute-in-memory (CIM) architectures that process and extract a variety of critical visual features in close physical proximity to where the data is stored in memory. The proposed research will embark on a uniquely integrated approach that addresses challenges at all levels, from devices, circuits, architectures, and algorithms leading to novel CV applications, inspired by neuroscience, such as low latency dynamic object classification, tracking and adaptive visual attention. The breadth of skillsets that are required to effectively train a new cadre of workforce in neuromorphic engineering for computer vision makes curriculum design and integration with existing frameworks incredibly challenging. The proposed BioVision educational consortium will address this issue. The main objective of this consortium is to collaborate and implement a comprehensive workforce development plan that incorporates evidence-based best practices to help train a new generation of engineers and researchers, who are equipped to satisfy the growing needs of the computer vision industry.The grand vision of this proposal is to reimagine modern computer vision (CV) pipelines that exist today and replace the components with bio-inspired sensors, processors and algorithms that can drastically improve energy efficiency, data efficiency and lower latency. To reinvent the CV pipeline, three research thrusts will be addressed simultaneously. Thrust 1 will focus on creating and building a new class of retina-inspired vision sensors, that outperforms existing cameras, such as frame-based or neuromorphic Dynamic Vision Sensors (DVS), in terms of features, efficiency and latency. Thrust 2 will focus on modeling, design and implementation of scalable corticomorphic networks on hardware, exhibiting non-linear neuromodulatory dynamics at multiple timescales using mixed-feedback control. Thrust 3 will focus on implementation of network architectures and algorithms inspired by neuroscience, such as reinforcement learning with stochastic rewards, event-based temporal pattern recognition. The proposed research has the potential to lead a generational shift in the fields of computer vision, neuromorphic computing, and artificial intelligence. Developing an energy-efficient event-based camera capable of versatile spatiotemporal pattern recognition and novel features inspired by the retina, along with a general purpose, programmable, event-based computer vision pipeline can have a transformative impact on our society, by impacting critical areas like healthcare, Internet of Things (IoT), military defense, edge computing and industrial automation. Enabling the use of advanced CV on personal electronics can revolutionize our lifestyle through technologies such as self-driving vehicles, always-on smart surveillance, and virtual/augmented reality (VR/AR) applications. Bio-inspired vision sensors, such as the DVS camera sold by companies like Prophesee and iniVation, are primarily developed in Europe and Asia and have no industry or academic contribution from USA. This proposal will address this national challenge by training a new generation of world-class researchers and provide the USA with a leading advantage in the deployment of next-generation computer vision systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.