SNNnow: Probabilistic Learning for Deep Spiking Neural Networks: Foundations and Hardware Co-Optimization

综述：深度神经网络(DNN)已经成为执行复杂学习任务的事实上的标准工具。DNN属于第二代人工神经网络(ANN)，它依赖神经元实现突触输入的无记忆非线性转换。受与大脑中神经元行为的生物类比的启发，第三代神经网络，也被称为尖峰神经网络(SNNS)，在90年代被引入。在SNN中，突触输入和神经元输出信号是棘波序列。该提案认为，使用SNN作为机器学习工具的时机已经到来，并提出了一种设计和实现SNN作为学习和推理机器的系统方法。智能优点：与ANN相比，SNN具有许多独特的优势：(I)它们是具有自然稀疏性的基于事件的系统，这有可能使深度学习机适用于能量有限的设备；(Ii)它们具有独特的本机能力来处理来自时间编码过程形式的数据，例如来自生物启发的传感器。这个项目的主要目标是建立一个理论框架来设计灵活的尖峰领域学习算法，这些算法是针对有监督和无监督认知任务的解决方案而定制的，以及它们在纳米级硬件结构上的共同优化。为此，本项目提出了一个基于有向信息图图形形式的原则性概率框架。广泛影响：本研究的结果预计将对越来越多的基于时间编码信号处理的实际应用产生深远影响，包括生物传感器和下一代通信系统，和/或需要采用与传统DNN相比具有显著较小功率预算的计算解决方案的实际应用。该研究方法基于多学科方法，融合了机器学习、信息论、概率图形模型、神经形态计算和纳米级的设备/系统架构。该机构的教育计划通过实践学习和实验活动，面向本科生和研究生。

Overview: Deep neural networks (DNN) have become the de-facto standard tool to carry out complex learningtasks. DNNs belong to the second generation of artificial neural networks (ANNs), which rely on neuronsthat implement memory-less non-linear transformations of the synaptic inputs. Motivated by the biologicalanalogy with the behavior of neurons in the brain, the third generation of neural networks, also referred toas Spiking Neural Networks (SNNs), was introduced in the nineties. In SNNs, synaptic input and neuronaloutput signals are spike trains. This proposal argues that the time for the use of SNNs as machine learningtools has come, and sets forth a systematic approach for the design and implementation of SNNs as learningand inference machines.Intellectual merit: SNNs have a number of unique advantages as compared to ANNs: (i) They are event-basedsystems with natural sparsity properties, which have the potential to make deep learning machines feasible forenergy-limited devices; (ii) They are uniquely capable to natively process data that comes in the form of timeencodedprocesses, for example, from bio-inspired sensors. The main goal of this project is the establishmentof a theoretical framework to enable the design of flexible spike-domain learning algorithms that are tailoredto the solution of supervised and unsupervised cognitive tasks, as well as their co-optimization on nanoscalehardware architectures. To this end, this project puts forth a principled probabilistic framework based on thegraphical formalism of Directed Information Graphs.Broader impact: The outcome of this research is expected to have a profound impact on the increasing numberof practical applications that are based on the processing of time-encoded signals, including biological sensorsand next-generation communication systems, and/or that require the adoption of computing solutions with asignificantly smaller power budget as compared to conventional DNNs. The research methodology is basedon a multi-disciplinary approach that integrates machine learning, information theory, probabilistic graphicalmodels, neuromorphic computing and device/system architecture at the nanoscale. The educational plan atthe home institution targets both undergraduate and graduate students via hands-on learning and experimentationactivities.