CAREER: Dynamic Distributed Learning in Spiking Neural Networks with Neural Architecture Search

如今，人工智能（AI）已经实现了大量的应用程序，从最新的聊天机器人到自动驾驶汽车，都能给你提供类似人类的问答体验。但是，所有这些人工智能的巨大成就在能源、内存和功耗方面都需要付出巨大的代价。在过去的十年中，峰值神经网络（snn）已经成为人工智能的低功耗替代品。SNN的主要吸引力在于它们提供了低功耗的架构实现，特别是对于算术运算。此外，与传统神经网络不同，snn随着时间和时间维度处理信息，如果利用得当，可以帮助下一代人工智能应用以更低的成本实现更好的性能和鲁棒性。然而，训练适合实际任务的snn一直是一个长期的挑战。该项目在基本优化策略上进行了创新，利用snn的时间特征产生具有多种连接和稀疏性的新架构，为分布式低功耗边缘计算应用带来显著的能效优势。此外，这项研究将支持博士和本科生的跨学科发展，并为培养下一代电气和计算机工程研究人员和实践者提供独特的教育基础设施。如今，为逼真的计算机视觉和相关任务部署大规模峰值神经网络（snn）是一个不小的挑战。该项目针对两个方向构建大规模SNN: 1)我们在神经架构搜索（NAS）上进行创新，以产生具有时间反馈连接的新SNN架构（这与传统的前馈深度学习网络形成鲜明对比）。2)我们使用snn特定的NAS优化在视觉任务的多个智能体上执行分布式学习，并展示了使用snn进行低功耗边缘计算的好处。特别是，我们开发了一种零射击方法，该方法不需要训练来搜索最佳网络架构，同时利用时间和空间稀疏性与修剪和相关技术。该策略预计将使SNN架构搜索的设计周期比现有工作缩短一到两个数量级。提议的NAS搜索将集成到一个联邦学习框架中，在这个框架中，具有不同资源和数据异质性的多个设备将一起学习。从本质上讲，这个项目发现新的SNN架构的框架可以产生强大而激进的解决方案，用于在具有极端资源限制的多种设备上学习，以启用大量分布式人工智能应用程序。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Artificial Intelligence (AI) has enabled a plethora of applications today, ranging from the most recent chatbots that give you a human-like question/answer experience to autonomous driving cars. But, all these massive feats with AI incur huge costs in terms of energy, memory, and power consumption. In the past decade, Spiking Neural Networks (SNNs) have emerged as a low-power alternative to AI. SNN’s main attraction lies in the fact that they offer low-power architectural implementations, especially for arithmetic operations. Furthermore, unlike traditional neural networks, SNNs process information over time and the temporal dimension, if leveraged suitably, can help enable the next generation of AI applications at lower cost with better performance and robustness. However, training SNNs suitably for realistic tasks has been a long-standing challenge. This project innovates on fundamental optimization strategies, using the temporal features in SNNs to yield new architectures with diverse connectivity and sparsity that yield significant energy-efficiency benefits for distributed low-power edge computing applications. Furthermore, this research will support the interdisciplinary development of a diverse cohort of Ph.D. and undergraduate students and provides a unique education infrastructure to train the next generation of electrical and computer engineering researchers and practitioners.Today, deploying large-scale spiking neural networks (SNNs) for realistic computer vision and related tasks is a non-trivial challenge. This project targets two directions to build large-scale SNNs: 1) We innovate on Neural Architecture Search (NAS) to yield new SNN architectures with temporal feedback connections (that is in stark contrast to conventional feedforward deep learning networks). 2) We use the SNN-specific NAS optimization to perform distributed learning on multiple agents for vision tasks and demonstrate the benefits of using SNNs for low-power edge computing. Particularly, we develop a zero-shot approach that does not require training to search for the optimal network architecture while leveraging temporal and spatial sparsity with pruning and related techniques. This strategy is expected to shorten the design cycle of SNN architecture search by one to two orders of magnitude over existing work. The proposed NAS search will be integrated into a federated learning framework where multiple devices with different resources and data heterogeneity are learning together. Essentially, this project’s framework for discovering new SNN architectures can yield powerful and radical solutions for learning on multiple devices with extreme resource limitations to enable numerous distributed AI applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.