Spike-Train Level Backpropagation for Training Deep Recurrent Spiking Neural Networks
Spike-Train Level Backpropagation for Training Deep Recurrent Spiking Neural Networks
复制标题
DOI：
--
复制
发表时间：
2019-08
期刊：
影响因子：
--
通讯作者：
Wenrui Zhang;Peng Li
中科院分区：
文献类型：
--
作者：
Wenrui Zhang;Peng Li
尖峰神经网络（SNN）很好地支持时空学习和节能的事件驱动硬件神经形态处理器。递归脉冲神经网络（RSNNs）是一类重要的神经网络，具有强大的计算能力。然而，RSNN的实际应用受到培训挑战的严重限制。生物启发的无监督学习在提高RSNN性能方面的能力有限。另一方面，现有的反向传播（BP）方法遭受高复杂性展开的时间，消失和爆炸梯度，以及近似分化的不连续尖峰活动时，应用到RSNN。为了在定义良好的损失函数下对RSNN进行监督训练，我们提出了一种新的Spike-Train级别RSNN反向传播（ST-RSBP）算法来训练深度RSNN。所提出的ST-RSBP直接计算在网络输出层定义的速率编码损失函数的梯度w.r.t可调参数。ST-RSBP的可扩展性是通过所提出的尖峰训练级计算来实现的，在此期间，SNN的时间效应在BP的前向和后向传递中被捕获。我们的ST-RSBP算法可以广泛应用于具有单个递归层的RSNN或具有多个前馈和递归层的深度RSNN。基于具有挑战性的语音和图像数据集，包括TI46，N-TIDIGITS，Fashion-MNIST和MNIST，ST-RSBP能够训练RSNN，其准确性超过当前最先进的SNN BP算法和传统的非尖峰深度学习模型。
Spiking neural networks (SNNs) well support spatiotemporal learning and energy-efficient event-driven hardware neuromorphic processors. As an important class of SNNs, recurrent spiking neural networks (RSNNs) possess great computational power. However, the practical application of RSNNs is severely limited by challenges in training. Biologically-inspired unsupervised learning has limited capability in boosting the performance of RSNNs. On the other hand, existing backpropagation (BP) methods suffer from high complexity of unrolling in time, vanishing and exploding gradients, and approximate differentiation of discontinuous spiking activities when applied to RSNNs. To enable supervised training of RSNNs under a well-defined loss function, we present a novel Spike-Train level RSNNs Backpropagation (ST-RSBP) algorithm for training deep RSNNs. The proposed ST-RSBP directly computes the gradient of a rated-coded loss function defined at the output layer of the network w.r.t tunable parameters. The scalability of ST-RSBP is achieved by the proposed spike-train level computation during which temporal effects of the SNN is captured in both the forward and backward pass of BP. Our ST-RSBP algorithm can be broadly applied to RSNNs with a single recurrent layer or deep RSNNs with multiple feed-forward and recurrent layers. Based upon challenging speech and image datasets including TI46, N-TIDIGITS, Fashion-MNIST and MNIST, ST-RSBP is able to train RSNNs with an accuracy surpassing that of the current state-of-art SNN BP algorithms and conventional non-spiking deep learning models.