CAREER: Dynamic Distributed Learning in Spiking Neural Networks with Neural Architecture Search

今天，人工智能(AI)已经实现了大量的应用，从最新的聊天机器人，为你提供类似人类的提问/回答体验，到自动驾驶汽车。但是，所有这些人工智能的巨大壮举在能源、内存和电力消耗方面都会招致巨大的成本。在过去的十年中，尖峰神经网络(SNN)已经成为人工智能的低功耗替代方案。SNN的主要吸引力在于它们提供了低功耗的体系结构实现，特别是对于算术运算。此外，与传统神经网络不同，SNN随着时间和时间维度处理信息，如果适当利用，可以帮助以更低的成本实现更好的性能和健壮性，从而实现下一代人工智能应用程序。然而，训练适合现实任务的SNN一直是一个长期存在的挑战。该项目在基本优化策略方面进行了创新，利用SNN中的时间特性来生成具有不同连接性和稀疏性的新体系结构，从而为分布式低功耗边缘计算应用程序带来显著的能效效益。此外，这项研究将支持不同的博士和本科生群体的跨学科发展，并提供独特的教育基础设施，以培养下一代电气和计算机工程研究人员和实践者。今天，为现实的计算机视觉和相关任务部署大规模棘波神经网络(SNN)是一个不平凡的挑战。本项目针对两个方向来构建大规模SNN：1)我们在神经结构搜索(NAS)方面进行了创新，提出了新的具有时间反馈连接的SNN结构(这与传统的前馈深度学习网络形成了鲜明的对比)。2)我们使用针对SNN的NAS优化在多个视觉任务的代理上进行分布式学习，并展示了使用SNN进行低功耗边缘计算的好处。特别是，我们开发了一种零镜头方法，该方法不需要训练来搜索最优的网络结构，同时利用剪枝和相关技术来利用时间和空间稀疏性。这一策略有望将SNN结构搜索的设计周期比现有工作缩短一到两个数量级。建议的NAS搜索将被集成到一个联合学习框架中，在该框架中，具有不同资源和数据异构性的多个设备将一起学习。从本质上讲，该项目发现新SNN架构的框架可以为在资源极端受限的多个设备上学习提供强大而激进的解决方案，以支持大量分布式人工智能应用程序。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Artificial Intelligence (AI) has enabled a plethora of applications today, ranging from the most recent chatbots that give you a human-like question/answer experience to autonomous driving cars. But, all these massive feats with AI incur huge costs in terms of energy, memory, and power consumption. In the past decade, Spiking Neural Networks (SNNs) have emerged as a low-power alternative to AI. SNN’s main attraction lies in the fact that they offer low-power architectural implementations, especially for arithmetic operations. Furthermore, unlike traditional neural networks, SNNs process information over time and the temporal dimension, if leveraged suitably, can help enable the next generation of AI applications at lower cost with better performance and robustness. However, training SNNs suitably for realistic tasks has been a long-standing challenge. This project innovates on fundamental optimization strategies, using the temporal features in SNNs to yield new architectures with diverse connectivity and sparsity that yield significant energy-efficiency benefits for distributed low-power edge computing applications. Furthermore, this research will support the interdisciplinary development of a diverse cohort of Ph.D. and undergraduate students and provides a unique education infrastructure to train the next generation of electrical and computer engineering researchers and practitioners.Today, deploying large-scale spiking neural networks (SNNs) for realistic computer vision and related tasks is a non-trivial challenge. This project targets two directions to build large-scale SNNs: 1) We innovate on Neural Architecture Search (NAS) to yield new SNN architectures with temporal feedback connections (that is in stark contrast to conventional feedforward deep learning networks). 2) We use the SNN-specific NAS optimization to perform distributed learning on multiple agents for vision tasks and demonstrate the benefits of using SNNs for low-power edge computing. Particularly, we develop a zero-shot approach that does not require training to search for the optimal network architecture while leveraging temporal and spatial sparsity with pruning and related techniques. This strategy is expected to shorten the design cycle of SNN architecture search by one to two orders of magnitude over existing work. The proposed NAS search will be integrated into a federated learning framework where multiple devices with different resources and data heterogeneity are learning together. Essentially, this project’s framework for discovering new SNN architectures can yield powerful and radical solutions for learning on multiple devices with extreme resource limitations to enable numerous distributed AI applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.