CAREER: Dynamic Distributed Learning in Spiking Neural Networks with Neural Architecture Search

人工智能（AI）今天已经启用了大量的应用程序，从最新的聊天机器人，为您提供类似人类的问题/答案体验到自动驾驶汽车。但是，所有这些人工智能的巨大成就都在能源、内存和功耗方面产生了巨大的成本。在过去的十年中，尖峰神经网络（SNN）已经成为人工智能的低功耗替代品。SNN的主要吸引力在于它们提供了低功耗的架构实现，特别是对于算术运算。此外，与传统神经网络不同，SNN随着时间的推移处理信息，如果适当利用时间维度，可以帮助以更低的成本实现下一代AI应用程序，并具有更好的性能和鲁棒性。然而，训练SNN适合现实任务一直是一个长期的挑战。该项目在基本优化策略上进行了创新，利用SNN中的时间特征产生具有不同连接性和稀疏性的新架构，为分布式低功耗边缘计算应用带来显著的能效优势。此外，这项研究将支持博士学位的多元化队列的跨学科发展。目前，为现实计算机视觉和相关任务部署大规模尖峰神经网络（SNN）是一项重大挑战。该项目针对构建大规模SNN的两个方向：1）我们在神经架构搜索（NAS）上进行创新，以产生具有时间反馈连接的新SNN架构（与传统的前馈深度学习网络形成鲜明对比）。2)我们使用SNN特定的NAS优化来对多个代理执行分布式学习，以执行视觉任务，并展示了使用SNN进行低功耗边缘计算的好处。特别是，我们开发了一种零拍摄方法，不需要训练来搜索最佳的网络架构，同时利用时间和空间稀疏性与修剪和相关技术。该策略有望将SNN架构搜索的设计周期缩短一到两个数量级。建议的NAS搜索将被集成到一个联合学习框架中，在这个框架中，具有不同资源和数据异构性的多个设备将一起学习。从本质上讲，该项目发现新SNN架构的框架可以产生强大而激进的解决方案，用于在极端资源限制的情况下在多个设备上学习，以实现众多分布式AI应用。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Artificial Intelligence (AI) has enabled a plethora of applications today, ranging from the most recent chatbots that give you a human-like question/answer experience to autonomous driving cars. But, all these massive feats with AI incur huge costs in terms of energy, memory, and power consumption. In the past decade, Spiking Neural Networks (SNNs) have emerged as a low-power alternative to AI. SNN’s main attraction lies in the fact that they offer low-power architectural implementations, especially for arithmetic operations. Furthermore, unlike traditional neural networks, SNNs process information over time and the temporal dimension, if leveraged suitably, can help enable the next generation of AI applications at lower cost with better performance and robustness. However, training SNNs suitably for realistic tasks has been a long-standing challenge. This project innovates on fundamental optimization strategies, using the temporal features in SNNs to yield new architectures with diverse connectivity and sparsity that yield significant energy-efficiency benefits for distributed low-power edge computing applications. Furthermore, this research will support the interdisciplinary development of a diverse cohort of Ph.D. and undergraduate students and provides a unique education infrastructure to train the next generation of electrical and computer engineering researchers and practitioners.Today, deploying large-scale spiking neural networks (SNNs) for realistic computer vision and related tasks is a non-trivial challenge. This project targets two directions to build large-scale SNNs: 1) We innovate on Neural Architecture Search (NAS) to yield new SNN architectures with temporal feedback connections (that is in stark contrast to conventional feedforward deep learning networks). 2) We use the SNN-specific NAS optimization to perform distributed learning on multiple agents for vision tasks and demonstrate the benefits of using SNNs for low-power edge computing. Particularly, we develop a zero-shot approach that does not require training to search for the optimal network architecture while leveraging temporal and spatial sparsity with pruning and related techniques. This strategy is expected to shorten the design cycle of SNN architecture search by one to two orders of magnitude over existing work. The proposed NAS search will be integrated into a federated learning framework where multiple devices with different resources and data heterogeneity are learning together. Essentially, this project’s framework for discovering new SNN architectures can yield powerful and radical solutions for learning on multiple devices with extreme resource limitations to enable numerous distributed AI applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.