CAREER: Reinventing Computer Vision through Bio-inspired Retinomorphic Vision Sensors, Corticomorphic Compute-In-Memory Processors and Event-based Algorithms

职业：通过仿生视网膜形态视觉传感器、皮质形态内存计算处理器和基于事件的算法重塑计算机视觉

• 批准号: 2338171
• 负责人: Rajkumar Chinnakonda Kubendran
• 金额: $ 54.98万
• 依托单位: University of Pittsburgh
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-05-01 至 2029-04-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2338171&HistoricalAwards=false
• 关键词: CAREER; Reinventing; Computer; Vision; through

State-of-the-art computer vision (CV) pipelines are compute/memory intensive and power hungry making them unsuitable for high-speed applications such as hypersonic missile tracking or resource-deficit edge applications such as autonomous drone navigation due to size, weight and power (SWaP) constraints. Neuromorphic engineering is a promising frontier to usher in the next generation of CV systems taking advantage of sparsity in the input and network architecture, reducing the number of operations through event-based computation i.e., compute only when necessary. This project aims to develop a versatile energy-efficient bio-inspired sensing, computing, and learning framework by developing a closely-knit system, from devices and circuits with rich spatio-temporal dynamics to network architectures inspired by the visual cortex and adaptive learning algorithms for visual perception. This will be achieved primarily using compute-in-memory (CIM) architectures that process and extract a variety of critical visual features in close physical proximity to where the data is stored in memory. The proposed research will embark on a uniquely integrated approach that addresses challenges at all levels, from devices, circuits, architectures, and algorithms leading to novel CV applications, inspired by neuroscience, such as low latency dynamic object classification, tracking and adaptive visual attention. The breadth of skillsets that are required to effectively train a new cadre of workforce in neuromorphic engineering for computer vision makes curriculum design and integration with existing frameworks incredibly challenging. The proposed BioVision educational consortium will address this issue. The main objective of this consortium is to collaborate and implement a comprehensive workforce development plan that incorporates evidence-based best practices to help train a new generation of engineers and researchers, who are equipped to satisfy the growing needs of the computer vision industry.The grand vision of this proposal is to reimagine modern computer vision (CV) pipelines that exist today and replace the components with bio-inspired sensors, processors and algorithms that can drastically improve energy efficiency, data efficiency and lower latency. To reinvent the CV pipeline, three research thrusts will be addressed simultaneously. Thrust 1 will focus on creating and building a new class of retina-inspired vision sensors, that outperforms existing cameras, such as frame-based or neuromorphic Dynamic Vision Sensors (DVS), in terms of features, efficiency and latency. Thrust 2 will focus on modeling, design and implementation of scalable corticomorphic networks on hardware, exhibiting non-linear neuromodulatory dynamics at multiple timescales using mixed-feedback control. Thrust 3 will focus on implementation of network architectures and algorithms inspired by neuroscience, such as reinforcement learning with stochastic rewards, event-based temporal pattern recognition. The proposed research has the potential to lead a generational shift in the fields of computer vision, neuromorphic computing, and artificial intelligence. Developing an energy-efficient event-based camera capable of versatile spatiotemporal pattern recognition and novel features inspired by the retina, along with a general purpose, programmable, event-based computer vision pipeline can have a transformative impact on our society, by impacting critical areas like healthcare, Internet of Things (IoT), military defense, edge computing and industrial automation. Enabling the use of advanced CV on personal electronics can revolutionize our lifestyle through technologies such as self-driving vehicles, always-on smart surveillance, and virtual/augmented reality (VR/AR) applications. Bio-inspired vision sensors, such as the DVS camera sold by companies like Prophesee and iniVation, are primarily developed in Europe and Asia and have no industry or academic contribution from USA. This proposal will address this national challenge by training a new generation of world-class researchers and provide the USA with a leading advantage in the deployment of next-generation computer vision systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

最先进的计算机视觉(CV)流水线是计算/内存密集型和耗电量大的，因此由于尺寸、重量和功率(交换)的限制，它们不适合高速应用，如高超声速导弹跟踪或资源匮乏的边缘应用，如自主无人机导航。神经形态工程是引入下一代CV系统的一个很有前途的前沿领域，它利用输入和网络体系结构的稀疏性，通过基于事件的计算减少操作数量，即只在必要时进行计算。该项目旨在通过开发一个紧密结合的系统，从具有丰富时空动力学的设备和电路到受视觉皮质启发的网络结构和视觉感知的自适应学习算法，来开发一个通用的节能生物灵感感知、计算和学习框架。这将主要使用内存中计算(CIM)架构来实现，该架构在物理上非常接近数据存储在内存中的位置来处理和提取各种关键的视觉特征。这项拟议的研究将在神经科学的启发下，着手采用一种独特的集成方法，从设备、电路、体系结构和算法到导致新的CV应用的所有级别的挑战，例如低延迟动态对象分类、跟踪和自适应视觉注意。在计算机视觉的神经形态工程领域有效培训新的劳动力所需的技能广度，使得课程设计和与现有框架的整合具有难以置信的挑战性。拟议的BioVision教育联盟将解决这一问题。该联盟的主要目标是合作并实施一项全面的劳动力发展计划，该计划纳入了基于证据的最佳实践，以帮助培训新一代工程师和研究人员，他们具备满足计算机视觉行业日益增长的需求的能力。该提议的宏伟愿景是重新想象现有的现代计算机视觉(CV)管道，并用生物启发的传感器、处理器和算法取代组件，这些组件可以显著提高能源效率、数据效率和更低的延迟。为了重塑简历管道，将同时解决三个研究推动力。推力1号将专注于创造和制造一种受视网膜启发的新型视觉传感器，在功能、效率和延迟方面优于现有的摄像头，如基于帧的或神经形态动态视觉传感器(DVS)。推力2将专注于在硬件上对可扩展的皮质共形网络进行建模、设计和实现，使用混合反馈控制在多个时间尺度上展示非线性神经调制动力学。Struts 3将专注于实施受神经科学启发的网络结构和算法，例如随机奖励强化学习、基于事件的时间模式识别。这项拟议的研究有可能在计算机视觉、神经形态计算和人工智能领域引领一代人的转变。开发一种能效高的基于事件的相机，能够进行多功能时空模式识别和受视网膜启发的新颖功能，以及通用、可编程、基于事件的计算机视觉管道，通过影响医疗保健、物联网(IoT)、军事防御、边缘计算和工业自动化等关键领域，可以对我们的社会产生革命性影响。在个人电子产品上使用先进的简历可以通过自动驾驶汽车、始终在线的智能监控和虚拟/增强现实(VR/AR)应用等技术来彻底改变我们的生活方式。仿生视觉传感器，如Prophesee和iniVation等公司销售的DVS相机，主要在欧洲和亚洲开发，没有来自美国的行业或学术贡献。这项提议将通过培养新一代世界级研究人员来应对这一国家挑战，并在部署下一代计算机视觉系统方面为美国提供领先优势。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。