Neural Behavior-Based Approach for Neural Network Pruning
Neural Behavior-Based Approach for Neural Network Pruning
复制标题
DOI：
10.1587/transinf.2019edp7177
复制
发表时间：
2020-05
期刊：
影响因子：
--
通讯作者：
Koji Kamma;Yuki Isoda;Sarimu Inoue;T. Wada
中科院分区：
文献类型：
--
作者：
Koji Kamma;Yuki Isoda;Sarimu Inoue;T. Wada
本文提出了一种减少训练神经网络模型的全连接层和卷积层冗余的方法。所提出的方法包括两个步骤，1）神经编码：通过由其对应于实际输入的输出组成的向量对每个神经元的行为进行编码;以及2）神经统一：统一具有相似行为向量的神经元。所提出的方法不是只修剪一个类似的神经元，而是让剩余的神经元模仿修剪后的神经元的行为。因此，所提出的方法可以减少神经元的数量，具有较小的准确性牺牲，而无需重新训练。我们的方法也可以用于压缩卷积层。在卷积层中，每个通道的行为由其输出特征图编码，并且其行为可以被其他通道很好地模仿的通道被修剪并更新剩余的权重。通过几个实验，我们证实了所提出的方法比现有的方法性能更好。
SUMMARY This paper presents a method for reducing the redundancy in both fully connected layers and convolutional layers of trained neural network models. The proposed method consists of two steps, 1) Neuro-Coding: to encode the behavior of each neuron by a vector composed of its outputs corresponding to actual inputs and 2) Neuro-Unification: to unify the neurons having the similar behavioral vectors. Instead of just pruning one of the similar neurons, the proposed method let the remaining neuron emulate the behavior of the pruned one. Therefore, the proposed method can reduce the number of neurons with small sacrifice of accuracy without retraining. Our method can be applied for compressing convolutional layers as well. In the convolutional layers, the behavior of each channel is encoded by its output feature maps, and channels whose behaviors can be well emulated by other channels are pruned and update the remaining weights. Through several experiments, we comfirmed that the proposed method performs better than the existing methods.