CMOS+X: Retinomorphic Infrared Imager with Sparsity-adaptive Machine-Learning Accelerator

实时信号分析在自主导航和机器人导航等应用中起着至关重要的作用。然而，传统的成像系统受到与将大量数据传输到集中式处理器相关的延迟和高能源成本的阻碍。为了克服这些缺点，该项目将在传感器阵列中直接嵌入计算元素，以便在本地运行机器学习算法，以提供实时和低功耗的对象识别和运动跟踪能力。这种本地传感器内计算方法将通过避免数据传输瓶颈来显著减少能源消耗和延迟。该项目将从生物视网膜获得灵感，共同优化有机红外传感器和硅电路的设计，生物视网膜对动态变化高度敏感，非常适合运动分析。与传统体系结构相比，拟议的原型有望降低高达100倍的能源消耗，从而为低功耗机器学习铺平道路。集成研究将有助于推进美国国内的半导体制造技术，并支持劳动力培训。研究团队还将开展外联活动，以促进人们对科学和工程领域各种职业道路的认识，以及工程职业的回报。该研究项目的目标是集成视网膜红外传感器和硅电路，以创建具有机器学习能力的成像系统原型，用于运动分析。该设计策略为有机传感器和硅电路分配了互补的角色：视网膜形传感器将在时间和空间域生成高度稀疏的特征提取数据，而电路将利用稀疏性来提高整体系统性能和节能。第一个研究目标集中在提高传感器的信号增益和调整时间常数，以将简化的数据传输到硅处理器。第二个目标是优化稀疏性自适应架构，使其能够处理广泛的稀疏性级别，并使用65 nm技术实现电路设计。第三个目标是建立将有机传感器阵列集成到硅芯片上的处理流程，并通过测量目标跟踪和分类的成功率来评估成像器的功能。最终的原型将为有效平衡智能红外成像系统中的能量使用、噪声和变化容差以及延迟的设计指南提供有价值的见解。拟议的成像仪配备了复杂而节能的机器学习能力，将在包括导航、生物医学成像、安全和机器视觉应用在内的各个领域具有广泛的适用性。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Real-time signal analysis plays a critical role in applications such as autonomous navigation and robotic guidance. However, conventional imaging systems are hindered by delays and high energy costs associated with transferring large amounts of data to centralized processors. To overcome those shortcomings, this project will embed computational elementals directly in the sensor arrays to locally run machine-learning algorithms that would provide capabilities for object recognition and motion tracking in real time and at low power. Such local in-sensor computing approach would offer significant reductions in energy consumption and delays by avoiding the data transfer bottlenecks. This project will co-optimize the designs of organic infrared sensors and silicon circuits, taking inspiration from the biological retina, which is highly sensitive to dynamic changes and well-suited for motion analysis. The proposed prototype is expected to reduce energy consumption by up to 100 times compared to conventional architectures, thereby paving the way for low-power machine learning. The integration research will contribute to advancing semiconductor manufacturing technologies within the United States and support workforce training. The research team will also engage in outreach activities to promote awareness of various career paths in science and engineering and the rewards of engineering careers.The goal of this research project is to integrate retinomorphic infrared sensors and silicon circuits in order to create a prototype imaging system with machine-learning capabilities for motion analysis. The design strategy assigns complementary roles to the organic sensors and silicon circuits: the retinomorphic sensors will generate highly sparse, feature-extracted data in both temporal and spatial domains, while the circuitries will use the sparsity to boost the overall system performance and power savings. The first research objective focuses on enhancing the sensor’s signal gain and adjusting the time constant to deliver streamlined data into the silicon processor. The second objective aims to optimize sparsity-adaptive architectures that can handle a wide range of sparsity levels and implement the circuit designs using 65 nm technology. The third objective involves establishing the processing workflow to integrate organic sensor arrays onto silicon chips and evaluating the functionalities of the imager by measuring the success rate of object tracking and classification. The resulting prototype will offer valuable insights into design guidelines for effectively balancing energy use, noise and variation tolerance, and latency in smart infrared imaging systems. The proposed imager, equipped with sophisticated yet energy-efficient machine learning capabilities, will have broad applicability across various fields including navigation, biomedical imaging, security, and machine vision applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.