A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
复制标题
DOI：
10.1007/978-3-030-01237-3_12
复制
发表时间：
2018-04
期刊：
影响因子：
--
通讯作者：
Tianyun Zhang;Shaokai Ye;Kaiqi Zhang;Jian Tang;Wujie Wen;M. Fardad;Yanzhi Wang
中科院分区：
文献类型：
--
作者：
Tianyun Zhang;Shaokai Ye;Kaiqi Zhang;Jian Tang;Wujie Wen;M. Fardad;Yanzhi Wang
最近已经研究了深度神经网络（DNN）的权重修剪方法，但该领域的先前工作主要是启发式的迭代修剪，从而缺乏对权重降低率和收敛时间的保证。为了减轻这些限制，我们提出了一个系统的权重修剪框架DNN使用交替方向乘法器（ADMM）。我们首先将DNN的权重修剪问题表示为具有指定稀疏性要求的组合约束的非凸优化问题，然后采用ADMM框架进行系统权重修剪。利用ADMM将原非凸优化问题分解为两个子问题，并迭代求解。其中一个子问题可以用随机梯度下降法求解，另一个子问题可以用解析法求解。此外，我们的方法实现了快速的收敛速度。权重修剪的结果是非常有希望的，并始终优于先前的工作。在MNIST数据集的LeNet-5模型上，我们实现了71.2倍的权重降低而没有精度损失。在ImageNet数据集的AlexNet模型上，我们实现了21倍的权重降低，而没有准确性损失。当我们专注于卷积层修剪以减少计算量时，与之前的工作相比，我们可以将总计算量减少五倍（实现卷积层的总重量减少13.4倍）。我们的模型和代码在https：//github上发布。com/KaiqiZhang/admm-pruning.
Weight pruning methods for deep neural networks (DNNs) have been investigated recently, but prior work in this area is mainly heuristic, iterative pruning, thereby lacking guarantees on the weight reduction ratio and convergence time. To mitigate these limitations, we present a systematic weight pruning framework of DNNs using the alternating direction method of multipliers (ADMM). We first formulate the weight pruning problem of DNNs as a nonconvex optimization problem with combinatorial constraints specifying the sparsity requirements, and then adopt the ADMM framework for systematic weight pruning. By using ADMM, the original nonconvex optimization problem is decomposed into two subproblems that are solved iteratively. One of these subproblems can be solved using stochastic gradient descent, the other can be solved analytically. Besides, our method achieves a fast convergence rate. The weight pruning results are very promising and consistently outperform the prior work. On the LeNet-5 model for the MNIST data set, we achieve 71.2 times weight reduction without accuracy loss. On the AlexNet model for the ImageNet data set, we achieve 21 times weight reduction without accuracy loss. When we focus on the convolutional layer pruning for computation reductions, we can reduce the total computation by five times compared with the prior work (achieving a total of 13.4 times weight reduction in convolutional layers). Our models and codes are released at https://github. com/KaiqiZhang/admm-pruning.