CMOS+X: Retinomorphic Infrared Imager with Sparsity-adaptive Machine-Learning Accelerator

实时信号分析在自主导航和机器人制导等应用中起着至关重要的作用。然而，传统的成像系统受到传输大量数据到中央处理器的延迟和高能源成本的阻碍。为了克服这些缺点，该项目将直接在传感器阵列中嵌入计算元素，以本地运行机器学习算法，从而提供实时、低功耗的目标识别和运动跟踪能力。这种局部传感器内计算方法通过避免数据传输瓶颈，可以显著降低能耗和延迟。该项目将从生物视网膜中获得灵感，共同优化有机红外传感器和硅电路的设计，生物视网膜对动态变化高度敏感，非常适合运动分析。与传统架构相比，该原型有望将能耗降低100倍，从而为低功耗机器学习铺平道路。集成研究将有助于推进半导体制造技术在美国和支持劳动力培训。研究小组亦会进行外展活动，以提高人们对科学和工程的各种职业道路以及工程职业的回报的认识。该研究项目的目标是集成视纯红外传感器和硅电路，以创建一个具有机器学习功能的原型成像系统，用于运动分析。该设计策略为有机传感器和硅电路分配了互补的角色：视纯传感器将在时间和空间域中生成高度稀疏的、特征提取的数据，而电路将利用这种稀疏性来提高整体系统性能和节能。第一个研究目标是提高传感器的信号增益和调整时间常数，以将流线型数据传输到硅处理器中。第二个目标是优化稀疏度自适应架构，该架构可以处理广泛的稀疏度级别，并使用65纳米技术实现电路设计。第三个目标涉及建立将有机传感器阵列集成到硅芯片上的处理工作流程，并通过测量目标跟踪和分类的成功率来评估成像仪的功能。由此产生的原型将为智能红外成像系统中有效平衡能源使用、噪音和变化容忍度以及延迟的设计指南提供有价值的见解。该成像仪配备了复杂而节能的机器学习功能，将在导航、生物医学成像、安全和机器视觉应用等各个领域具有广泛的适用性。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Real-time signal analysis plays a critical role in applications such as autonomous navigation and robotic guidance. However, conventional imaging systems are hindered by delays and high energy costs associated with transferring large amounts of data to centralized processors. To overcome those shortcomings, this project will embed computational elementals directly in the sensor arrays to locally run machine-learning algorithms that would provide capabilities for object recognition and motion tracking in real time and at low power. Such local in-sensor computing approach would offer significant reductions in energy consumption and delays by avoiding the data transfer bottlenecks. This project will co-optimize the designs of organic infrared sensors and silicon circuits, taking inspiration from the biological retina, which is highly sensitive to dynamic changes and well-suited for motion analysis. The proposed prototype is expected to reduce energy consumption by up to 100 times compared to conventional architectures, thereby paving the way for low-power machine learning. The integration research will contribute to advancing semiconductor manufacturing technologies within the United States and support workforce training. The research team will also engage in outreach activities to promote awareness of various career paths in science and engineering and the rewards of engineering careers.The goal of this research project is to integrate retinomorphic infrared sensors and silicon circuits in order to create a prototype imaging system with machine-learning capabilities for motion analysis. The design strategy assigns complementary roles to the organic sensors and silicon circuits: the retinomorphic sensors will generate highly sparse, feature-extracted data in both temporal and spatial domains, while the circuitries will use the sparsity to boost the overall system performance and power savings. The first research objective focuses on enhancing the sensor’s signal gain and adjusting the time constant to deliver streamlined data into the silicon processor. The second objective aims to optimize sparsity-adaptive architectures that can handle a wide range of sparsity levels and implement the circuit designs using 65 nm technology. The third objective involves establishing the processing workflow to integrate organic sensor arrays onto silicon chips and evaluating the functionalities of the imager by measuring the success rate of object tracking and classification. The resulting prototype will offer valuable insights into design guidelines for effectively balancing energy use, noise and variation tolerance, and latency in smart infrared imaging systems. The proposed imager, equipped with sophisticated yet energy-efficient machine learning capabilities, will have broad applicability across various fields including navigation, biomedical imaging, security, and machine vision applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.