Collaborative Research: FET: Medium: Energy-Efficient Persistent Learning-in-Memory with Quantum Tunneling Dynamic Synapses

本研究项目研究了一个框架，该框架可以显著提高使用基于量子隧道动态模拟存储器（DAM）设备的电路和系统架构训练人工智能（AI）系统的能源效率。2019年，训练一个顶级人工智能系统所需的能源超过了五辆美国汽车在其整个生命周期内运行所需的能源。从那以后，训练大规模人工智能系统的能源需求变得越来越糟糕，已经到了不可持续的地步。拟议的研究旨在开发新的学习硬件，使机器学习和人工智能系统的训练更具能源可持续性。该项目还在开发软件工具，用于培训可被研究界传播和采用的人工智能系统。该项目正在开发的新型在线学习和记忆巩固算法将通过圣地亚哥超级计算机中心的神经科学门户（NSG）门户与一个公开共享的通用神经形态认知计算平台集成。该项目与Efabless公司合作，支持混合信号集成电路（IC）设计工具的开源开发，这些工具正在通过课堂教学和项目进行评估。这项研究项目的技术活动是基于一种超节能的突触元件，称为Fowler-Nordheim动态模拟存储器（FN-DAM），可以很容易地在标准集成电路工艺上制造。突触元件的记忆保持特性先前已被证明是自适应的，并且可以与突触更新所需的能量相权衡。这些FN-DAM特性是在以下研究目标的背景下探索的：1)研究新的基于FN-DAM的神经网络训练和学习算法和架构：正在探索将FN-DAM阵列的动态与标准卷积神经网络的训练公式联系起来的机制。有效的一次性连续在线学习技术正在被研究，利用FN-DAM的动力学来提高学习的速度和鲁棒性。该框架被用于探索基于FN-DAM的结构与结合了情景记忆和增量学习范式的神经形态记忆结构之间的联系；2)基于FN-DAM的新型内存计算和片上学习架构的研究：正在研究将FN-DAM阵列与CMOS计算电路以及片上适应和学习策略集成在一起的模拟内存计算学习架构；3)基于FN-DAM的软硬件协同设计框架的验证：该项目正在验证协同设计框架，以实现神经网络训练的高能效，该框架使用由加州大学圣地亚哥分校（UCSD）开发和维护的大规模神经形态认知计算的NSF CISE社区研究基础设施（CRI）。该项目还验证了使用标准集成电路工艺制造的原型可以实现的能效改进。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This research project investigates a framework that can significantly improve the energy-efficiency of training artificial intelligence (AI) systems using circuits and system architectures that are based on quantum-tunneling dynamic-analog-memory (DAM) devices. In 2019, the energy required to train a top-of-the-line AI system was more than the energy required to operate five US cars over their entire lifetime. The energy requirements for training large-scale AI systems have only gotten worse since to the point of being unsustainable. The proposed research aims to develop novel learning hardware that will make the training of ML and AI systems more energy sustainable. The project is also developing software tools for training AI systems that can be disseminated and adopted by the research community. The novel online learning and memory consolidation algorithms that are being developed in this project will be integrated with an openly shared, general-purpose neuromorphic cognitive computing platform available through the Neuroscience Gateway (NSG) Portal at the San Diego Supercomputer Center. In collaboration with Efabless Inc. the project is supporting open-source development of mixed-signal integrated circuits (IC) design tools that is being evaluated through in class-room instruction and projects.The technical activities of this research project are based on an ultra-energy-efficient synaptic element called Fowler-Nordheim Dynamic Analog Memory (FN-DAM) that can be easily fabricated on a standard integrated circuits process. The memory retention property of the synaptic element has been previously shown to be adaptive and can be traded-off with the energy required for synaptic updates. These FN-DAM properties are being explored within the context of the following research objectives: 1) Investigation into novel FN-DAM based neural network training and learning algorithms and architecture: Mechanisms are being explored that can connect the dynamics of FN-DAM array with the training formulations of standard convolutional neural network. Efficient one-shot continual online learning techniques are being investigated that exploit the dynamics of FN-DAM to improve the speed and robustness of learning. The framework is being used to explore connections between the FN-DAM based architectures with neuromorphic memory architectures that combines episodic-memories with incremental learning paradigms; 2) Investigation into novel FN-DAM based compute-in-memory and on-chip learning architectures: Analog compute-in-memory learning architectures are being investigated that integrate FN-DAM arrays with CMOS computing circuits and on-chip adaptation and learning strategies; 3) Validation of the FN-DAM based hardware-software co-design framework: The project is validating the co-design framework for achieving high energy-efficiency in neural network training using the NSF CISE Community Research Infrastructure (CRI) for large-scale neuromorphic cognitive computing developed and maintained at University of California at San Diego (UCSD). The project is also validating the energy-efficiency improvements that can be achieved using prototypes that will be fabricated in a standard integrated circuits process.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.