Scaling Up Exact Neural Network Compression by ReLU Stability
Scaling Up Exact Neural Network Compression by ReLU Stability
复制标题
DOI：
--
复制
发表时间：
2021-02
期刊：
影响因子：
--
通讯作者：
Thiago Serra;Abhinav Kumar;Srikumar Ramalingam
中科院分区：
文献类型：
--
作者：
Thiago Serra;Abhinav Kumar;Srikumar Ramalingam
我们可以压缩整流器网络，同时精确地保留其关于给定输入域的底层功能，如果它的一些神经元是稳定的。然而，目前确定具有整流线性单元（ReLU）激活的神经元的稳定性的方法需要解决或找到多个离散优化问题的良好近似。在这项工作中，我们介绍了一个算法的基础上解决一个单一的优化问题，以确定所有稳定的神经元。我们的方法比CIFAR-10上的最先进方法快183倍，这使我们能够在几分钟内探索更深（5 x 100）和更宽（2 x 800）网络的精确压缩。对于在一定量的L1正则化下训练的分类器，我们可以在CIFAR-10数据集上删除多达56%的连接。代码可在以下链接https://github.com/yuxwind/ExactCompression上获得。
We can compress a rectifier network while exactly preserving its underlying functionality with respect to a given input domain if some of its neurons are stable. However, current approaches to determine the stability of neurons with Rectified Linear Unit (ReLU) activations require solving or finding a good approximation to multiple discrete optimization problems. In this work, we introduce an algorithm based on solving a single optimization problem to identify all stable neurons. Our approach is on median 183 times faster than the state-of-art method on CIFAR-10, which allows us to explore exact compression on deeper (5 x 100) and wider (2 x 800) networks within minutes. For classifiers trained under an amount of L1 regularization that does not worsen accuracy, we can remove up to 56% of the connections on the CIFAR-10 dataset. The code is available at the following link, https://github.com/yuxwind/ExactCompression.