StructADMM: Achieving Ultrahigh Efficiency in Structured Pruning for DNNs
StructADMM: Achieving Ultrahigh Efficiency in Structured Pruning for DNNs
复制标题
DOI：
10.1109/tnnls.2020.3045153
复制
发表时间：
2021-02-12
影响因子：
10.4
通讯作者：
Wang, Yanzhi
中科院分区：
文献类型：
--
作者：
Zhang, Tianyun;Ye, Shaokai;Wang, Yanzhi
已经证明了深神经网络（DNN）的重量修剪方法可以达到良好的模型修剪速率而不会损失准确性，从而减轻了大规模DNN的重大计算/存储要求。已经提出了结构化的修剪方法来克服不规则网络结构的局限性，并证明了实际的GPU加速度。但是，在先前的工作中，需要保持准确性时，修剪率（稀疏度）和GPU加速度受到限制（少于50％）。在这项工作中，我们通过提出一个针对DNN的结构重量修剪的统一的系统框架来克服这些局限性。它是一个可用于诱导不同类型的结构化稀疏性的框架，例如FelterWise，Channelwise和Shapewise稀疏性以及非结构性的稀疏性。提出的框架将随机梯度下降（SGD； ADAM）与交替的乘数（ADMM）方法结合在一起，并且可以理解为一种动态正则化方法，其中在每种迭代中分析了正则化目标。利用ADMM的特殊特征，我们进一步提出了一个进行性的，多步重的修剪框架和网络纯化和未使用的路径去除程序，以实现较高的修剪率而不会准确损失。在Alexnet模型上的准确性损失的情况下，我们在两个GPU上达到了2.58倍和3.65倍的平均测量速度，显然表现优于先前的工作。当中等准确度损失2％时，平均加速度达到3.15倍和8.52倍。在这种情况下，卷积层的模型压缩为15.0倍，对应于11.93倍的CPU加速度。作为另一个示例，对于CIFAR-10数据集的RESNET-18模型，我们在Conv层上实现了前所未有的54.2倍结构化修剪率。与最近的工作相比，这比原始的640移动GPU上的修剪率高32倍，并且与原始未经修复的DNN模型相比，可以进一步转化为7.6倍推理时间加速。我们在链接http://bit.ly/2m0v7do上共享我们的代码和模型。
Weight pruning methods of deep neural networks (DNNs) have been demonstrated to achieve a good model pruning rate without loss of accuracy, thereby alleviating the significant computation/storage requirements of large-scale DNNs. Structured weight pruning methods have been proposed to overcome the limitation of irregular network structure and demonstrated actual GPU acceleration. However, in prior work, the pruning rate (degree of sparsity) and GPU acceleration are limited (to less than 50%) when accuracy needs to be maintained. In this work, we overcome these limitations by proposing a unified, systematic framework of structured weight pruning for DNNs. It is a framework that can be used to induce different types of structured sparsity, such as filterwise, channelwise, and shapewise sparsity, as well as nonstructured sparsity. The proposed framework incorporates stochastic gradient descent (SGD; or ADAM) with alternating direction method of multipliers (ADMM) and can be understood as a dynamic regularization method in which the regularization target is analytically updated in each iteration. Leveraging special characteristics of ADMM, we further propose a progressive, multistep weight pruning framework and a network purification and unused path removal procedure, in order to achieve higher pruning rate without accuracy loss. Without loss of accuracy on the AlexNet model, we achieve 2.58x and 3.65x average measured speedup on two GPUs, clearly outperforming the prior work. The average speedups reach 3.15x and 8.52x when allowing a moderate accuracy loss of 2%. In this case, the model compression for convolutional layers is 15.0x, corresponding to 11.93x measured CPU speedup. As another example, for the ResNet-18 model on the CIFAR-10 data set, we achieve an unprecedented 54.2x structured pruning rate on CONV layers. This is 32x higher pruning rate compared with recent work and can further translate into 7.6x inference time speedup on the Adreno 640 mobile GPU compared with the original, unpruned DNN model. We share our codes and models at the link http://bit.ly/2M0V7DO.