Synaptic Stripping: How Pruning Can Bring Dead Neurons Back to Life
Synaptic Stripping: How Pruning Can Bring Dead Neurons Back to Life
复制标题
突触剥离：修剪如何使死亡的神经元起死回生
DOI：
10.1109/ijcnn54540.2023.10191397
复制
发表时间：
2023
期刊：
影响因子：
--
通讯作者：
L. D. Whitley
中科院分区：
文献类型：
--
作者：
Tim Whitaker;L. D. Whitley
整流线性单元(REU)是深度神经网络中激活函数的默认选择。虽然REU激活表现出出色的经验表现，但它们可能成为死亡神经元问题的牺牲品。在这些情况下，输入到神经元的权重最终被推入一种状态，即神经元对所有输入输出为零。因此，对于所有的输入，梯度也是零，这意味着输入神经元的权重不能更新。神经元不能从直接反向传播中恢复，模型容量降低，因为这些参数不能再进一步优化。受同名神经学过程的启发，我们引入突触剥离作为对抗这个死亡神经元问题的一种手段。通过在训练过程中自动删除有问题的连接，我们可以再生死亡神经元，并显著提高模型容量和参数利用率。突触剥离很容易实现，产生的稀疏网络比从中派生出的密集网络更有效。我们进行了几项消融研究，以研究这些动力学作为网络宽度和深度的函数，并在各种基准数据集上使用Vision Transformers进行了突触剥离的探索。
Rectified Linear Units (ReLU) are the default choice for activation functions in deep neural networks. While they demonstrate excellent empirical performance, ReLU activations can fall victim to the dead neuron problem. In these cases, the weights feeding into a neuron end up being pushed into a state where the neuron outputs zero for all inputs. Consequently, the gradient is also zero for all inputs, which means that the weights which feed into the neuron cannot update. The neuron is not able to recover from direct back propagation and model capacity is reduced as those parameters can no longer be further optimized. Inspired by a neurological process of the same name, we introduce Synaptic Stripping as a means to combat this dead neuron problem. By automatically removing problematic connections during training, we can regenerate dead neurons and significantly improve model capacity and parametric utilization. Synaptic Stripping is easy to implement and results in sparse networks that are more efficient than the dense networks they are derived from. We conduct several ablation studies to investigate these dynamics as a function of network width and depth and we conduct an exploration of Synaptic Stripping with Vision Transformers on a variety of benchmark datasets.
DOI：
--
发表时间：
2020-06
期刊：
ArXiv
影响因子：
--
作者：
Hidenori Tanaka;D. Kunin;Daniel L. K. Yamins;S. Ganguli
通讯作者：
Hidenori Tanaka;D. Kunin;Daniel L. K. Yamins;S. Ganguli
深度（呃）学习。
DOI：
10.1523/jneurosci.0153-18.2018
发表时间：
2018
期刊：
The Journal of neuroscience : the official journal of the Society for Neuroscience
影响因子：
--
作者：
Srinivasan,Shyam;Greenspan,RalphJ;Stevens,CharlesF;Grover,Dhruv
通讯作者：
Grover,Dhruv