DropOut and DropConnect for Reliable Neuromorphic Inference under Energy and Bandwidth Constraints in Network Connectivity
DropOut and DropConnect for Reliable Neuromorphic Inference under Energy and Bandwidth Constraints in Network Connectivity
复制标题
DropOut 和 DropConnect 在网络连接的能量和带宽约束下实现可靠的神经形态推理
DOI：
--
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
G. Cauwenberghs
中科院分区：
文献类型：
--
作者：
Yasufumi Sakai;Bruno U. Pedroni;Siddharth Joshi;Abraham Akinin;G. Cauwenberghs
辍学和DropConnect被称为有效方法，可以通过删除神经单位的状态或在整个训练过程中每个时间实例中随机选择的突触连接的权重来改善神经网络的概括性能。在本文中，我们扩展了这些方法在神经形态尖峰神经网络（SNN）硬件的设计中的使用，以进一步提高推理的可靠性，这受到资源约束网络连接错误的影响。这种能量和带宽的限制是在神经单元之间的通信中出现低功率操作的，这会导致由于传输中的超时错误而导致的尖峰事件。网络训练期间的辍学和连接过程与推理期间网络的统计模型对齐，该模型在神经状态和突触连接的传播中说明了这些随机误差。因此，在训练过程中使用辍学和dropconnect可以同时满足两个设计目标：最大化带宽，同时最大程度地减少神经形态硬件的推理能量。 MNIST任务上的5层完全连接的784-500-500-500-10 SNN的模型模拟显示在推理期间的带宽5倍和10倍以大于98％的精度，使用辍学和DropConnect在反向传播培训期间分别。
DropOut and DropConnect are known as effective methods to improve on the generalization performance of neural networks, by either dropping states of neural units or dropping weights of synaptic connections randomly selected at each time instance throughout the training process. In this paper, we extend on the use of these methods in the design of neuromorphic spiking neural networks (SNN) hardware to improve further on the reliability of inference as impacted by resource constrained errors in network connectivity. Such energy and bandwidth constraints arise for low-power operation in the communication between neural units, which cause dropped spike events due to timeout errors in the transmission. The DropOut and DropConnect processes during training of the network are aligned with a statistical model of the network during inference that accounts for these random errors in the transmission of neural states and synaptic connections. The use of DropOut and DropConnect during training hence allows to simultaneously meet two design objectives: maximizing bandwidth, while minimizing energy of inference in neuromorphic hardware. Simulations of the model with a 5-layer fully connected 784-500-500-500-10 SNN on the MNIST task show a 5-fold and 10-fold improvement in bandwidth during inference at greater than 98% accuracy, using DropOut and DropConnect respectively during backpropagation training.