CAREER: Dynamic Distributed Learning in Spiking Neural Networks with Neural Architecture Search

职业：具有神经架构搜索的尖峰神经网络中的动态分布式学习

• 批准号: 2238227
• 负责人: Priyadarshini Panda
• 金额: $ 50.48万
• 依托单位: Yale University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-01-01 至 2027-12-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2238227&HistoricalAwards=false
• 关键词: CAREER; Dynamic; Distributed; Learning; Spiking

Artificial Intelligence (AI) has enabled a plethora of applications today, ranging from the most recent chatbots that give you a human-like question/answer experience to autonomous driving cars. But, all these massive feats with AI incur huge costs in terms of energy, memory, and power consumption. In the past decade, Spiking Neural Networks (SNNs) have emerged as a low-power alternative to AI. SNN’s main attraction lies in the fact that they offer low-power architectural implementations, especially for arithmetic operations. Furthermore, unlike traditional neural networks, SNNs process information over time and the temporal dimension, if leveraged suitably, can help enable the next generation of AI applications at lower cost with better performance and robustness. However, training SNNs suitably for realistic tasks has been a long-standing challenge. This project innovates on fundamental optimization strategies, using the temporal features in SNNs to yield new architectures with diverse connectivity and sparsity that yield significant energy-efficiency benefits for distributed low-power edge computing applications. Furthermore, this research will support the interdisciplinary development of a diverse cohort of Ph.D. and undergraduate students and provides a unique education infrastructure to train the next generation of electrical and computer engineering researchers and practitioners.Today, deploying large-scale spiking neural networks (SNNs) for realistic computer vision and related tasks is a non-trivial challenge. This project targets two directions to build large-scale SNNs: 1) We innovate on Neural Architecture Search (NAS) to yield new SNN architectures with temporal feedback connections (that is in stark contrast to conventional feedforward deep learning networks). 2) We use the SNN-specific NAS optimization to perform distributed learning on multiple agents for vision tasks and demonstrate the benefits of using SNNs for low-power edge computing. Particularly, we develop a zero-shot approach that does not require training to search for the optimal network architecture while leveraging temporal and spatial sparsity with pruning and related techniques. This strategy is expected to shorten the design cycle of SNN architecture search by one to two orders of magnitude over existing work. The proposed NAS search will be integrated into a federated learning framework where multiple devices with different resources and data heterogeneity are learning together. Essentially, this project’s framework for discovering new SNN architectures can yield powerful and radical solutions for learning on multiple devices with extreme resource limitations to enable numerous distributed AI applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

人工智能（AI）今天启用了大量应用程序，从最新的聊天机器人为您提供类似人类的问答体验到自动驾驶汽车。但是，所有这些庞大的壮举都以人工智能在能源，记忆和功耗方面产生了巨大的成本。在过去的十年中，尖峰神经网络（SNN）已成为AI的低功耗替代品。 SNN的主要吸引力在于它们提供了低功率建筑实现，尤其是用于算术操作。此外，与传统的神经网络不同，SNNS随着时间的流逝而处理信息，如果合适地利用了临时维度，则可以帮助下一代AI应用程序以较低的成本和更高的性能和稳健性。但是，适当地培训SNN来实现现实任务一直是一个长期的挑战。该项目对基本优化策略的创新，使用SNN中的临时功能，产生具有多样性连通性和稀疏性的新体系结构，可为分布式的低功率边缘计算应用带来显着的能效收益。此外，这项研究将支持各种博士学位的跨学科发展。和本科生，并提供了独特的教育基础设施，以培训下一代电气和计算机工程研究人员和从业人员。建筑，部署大型尖峰神经网络（SNNS）来实现现实的计算机视觉和相关任务，这是一项非琐事的挑战。该项目针对两个方向来构建大型SNNS：1）我们在神经体系结构搜索（NAS）上无效，以产生具有临时反馈连接的新SNN体系结构（与常规的进料深度学习网络形成鲜明对比）。 2）我们使用SNN特异性的NAS优化对视力任务进行分布式学习，并证明将SNN用于低功率边缘计算的好处。特别是，我们开发了一种零拍的方法，该方法不需要培训来搜索最佳网络体系结构，同时利用修剪和相关技术利用临时和空间稀疏性。预计该策略将在现有工作中缩短SNN体系结构搜索的设计周期一到两个数量级。拟议的NAS搜索将集成到一个联合学习框架中，其中具有不同资源和数据异质性的多个设备一起学习。从本质上讲，该项目用于发现新的SNN体系结构的框架可以产生强大而激进的解决方案，以在具有极端资源限制的多个设备上学习，以实现许多分布式AI应用程序。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子和更广泛影响的审查标准来通过评估来通过评估来获得支持的。