CAREER: Rethinking Spiking Neural Networks from a Dynamical System Perspective

神经形态计算算法正在成为推动机器学习研究的颠覆性范式。尽管这种受大脑启发的系统由于事件驱动的网络操作而实现了显着的节能，但神经形态尖峰神经网络（SNN）在很大程度上仍然限于静态视觉任务和卷积架构。因此，有一个未满足的需要，重新审视可扩展的SNN训练算法，从地面上锻造更强的相关性与生物相容性，以利用巨大的潜力，基于时间的信息处理和本地学习能力的SNN的顺序任务。该项目将尖峰架构视为事件驱动的动态系统，其中通过向平衡状态的收敛进行学习。神经元集体调整自己的配置（根据输入到神经网络系统的感觉输入），使它们能够更好地预测输入数据的想法一直是一个流行的假设。集体神经元状态可以被解释为对输入数据的解释。这一引人注目的中心思想为这项研究和教育计划提供了动力，它通过追求两种最近出现的训练神经架构的方法，即平衡传播（EP）和平衡上的隐式微分（IDE），它们相互之间具有很强的协同作用。该研究对人工智能（AI）和半导体行业以及整个社会都产生了深远的影响，其中颠覆性计算范例，如神经形态计算，新兴设备技术和跨层优化，与最先进的方法相比，可能会在数据密集型机器学习工作负载方面取得显着改善。 该项目将考虑一个综合的研究，教育和推广计划，考虑跨学科课程开发，研究生和本科生的研究指导，K-12的参与，在线教育模块的开发和提高少数民族的研究参与，以培养下一代的研究人员和工程师共同在机器学习和纳米电子学领域。最终研究议程有可能通过追求多学科视角-将机器学习和动态系统的见解与硬件结合起来，实现人工智能平台效率的飞跃。该项目涵盖以下主要领域的互补和跨学科探索：（1）通过将EP与现代Hopfield网络集成以实现注意力机制，在SNN中实现复杂任务的局部学习，（2）使用IDE为Spiking语言模型开发可扩展且计算高效的训练方法，（3）跨层软件-硬件-应用优化，用于在神经形态平台上有效实现算法创新，以实现大规模顺序学习任务。该项目的跨层性质，包括机器学习，动态系统建模，尖端人工智能应用和硬件设计，将成为追求跨学科劳动力发展计划的理想平台。如果成功的话，这项研究有可能开发出可扩展的，强大的，功率和能量高效的神经形态计算范例，适用于广泛的顺序处理任务，与传统深度学习解决方案（如大型语言模型）的巨大计算需求相比，这是一个重大转变。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准。

Neuromorphic computing algorithms are emerging to be a disruptive paradigm driving machine learning research. Despite the significant energy savings enabled by such brain-inspired systems due to event-driven network operation, neuromorphic spiking neural networks (SNNs) remain largely limited to static vision tasks and convolutional architectures. Hence, there is an unmet need to revisit scalable SNN training algorithms from the ground-up by forging stronger correlations with bio-plausibility to leverage the enormous potential of time-based information processing and local learning capability of SNNs for sequential tasks. The project approaches spiking architectures as event-driven dynamical systems, wherein learning occurs through the convergence towards equilibrium states. The idea that neurons collectively adjust themselves to configurations (according to the sensory input being fed into a neural network system) such that they can better predict the input data has been a popular hypothesis. The collective neuron states can be interpreted as explanations of the input data. This compelling central idea provides motivation for this research and education program by pursuing two recently emerging methodologies for training neural architectures viz - Equilibrium Propagation (EP) and Implicit Differentiation on Equilibrium (IDE) that bear strong synergies with each other. The research has far-reaching impacts on Artificial Intelligence (AI) and the semiconductor industry, and on society at large, where disruptive computing paradigms like neuromorphic computing, emerging device technologies and cross-layer optimizations can potentially achieve significant improvements in data-intensive machine learning workloads in contrast to state-of-the-art approaches. The project will consider an integrated research, education and outreach plan that considers interdisciplinary curriculum development, graduate and undergraduate research mentoring, K-12 involvement, online educational module development and enhancing minority research participation to train the next generation of researchers and engineers jointly in the fields of Machine Learning and Nanoelectronics.The presented end-to-end research agenda has the potential of enabling a quantum leap in the efficiency of AI platforms by pursuing a multi-disciplinary perspective -- combining insights from machine learning and dynamical systems to hardware. The project spans complementary and inter-twined explorations across the following thrust areas: (1) Enabling local learning in SNNs for complex tasks by integrating EP with modern Hopfield networks to implement attention mechanisms, (2) Using IDE for developing a scalable and computationally efficient training method for Spiking Language Models, (3) Cross-layer software-hardware-application optimizations for efficient implementation of the algorithmic innovations on neuromorphic platforms for large-scale sequential learning tasks. The cross-layer nature of the project ranging from machine learning, dynamical system modelling, cutting edge AI applications and hardware design will serve as an ideal platform to pursue an interdisciplinary workforce development program. If successful, the research has the potential of developing scalable, robust, power and energy efficient neuromorphic computing paradigms that are applicable to a broad range of sequential processing tasks - a significant shift from the huge computational requirements of conventional deep learning solutions like Large Language Models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.