SNIP: Single-shot Network Pruning based on Connection Sensitivity
SNIP: Single-shot Network Pruning based on Connection Sensitivity
复制标题
DOI：
--
复制
发表时间：
2018-09
期刊：
影响因子：
--
通讯作者：
Namhoon Lee;Thalaiyasingam Ajanthan;Philip H. S. Torr
中科院分区：
文献类型：
--
作者：
Namhoon Lee;Thalaiyasingam Ajanthan;Philip H. S. Torr
修剪大型神经网络，同时保持其性能往往是可取的，因为减少了空间和时间的复杂性。在现有的方法中，修剪是在迭代优化过程中完成的，其中使用的是按时间顺序设计的修剪时间表或附加的超参数，这削弱了它们的效用。在这项工作中，我们提出了一种新的方法，在训练之前的初始化阶段修剪给定的网络。为了实现这一目标，我们引入了一个显着性标准的基础上连接的敏感性，确定结构上重要的连接在网络中的给定任务。这消除了对预训练和复杂修剪时间表的需要，同时使其对架构变化具有鲁棒性。修剪后，稀疏网络以标准方式进行训练。我们的方法在MNIST、CIFAR-10和Tiny-ImageNet分类任务上获得了与参考网络几乎相同的精度，并且广泛适用于各种架构，包括卷积、残差和递归网络。与现有的方法不同，我们的方法使我们能够证明保留的连接确实与给定的任务相关。
Pruning large neural networks while maintaining their performance is often desirable due to the reduced space and time complexity. In existing methods, pruning is done within an iterative optimization procedure with either heuristically designed pruning schedules or additional hyperparameters, undermining their utility. In this work, we present a new approach that prunes a given network once at initialization prior to training. To achieve this, we introduce a saliency criterion based on connection sensitivity that identifies structurally important connections in the network for the given task. This eliminates the need for both pretraining and the complex pruning schedule while making it robust to architecture variations. After pruning, the sparse network is trained in the standard way. Our method obtains extremely sparse networks with virtually the same accuracy as the reference network on the MNIST, CIFAR-10, and Tiny-ImageNet classification tasks and is broadly applicable to various architectures including convolutional, residual and recurrent networks. Unlike existing methods, our approach enables us to demonstrate that the retained connections are indeed relevant to the given task.