ClickTrain: efficient and accurate end-to-end deep learning training via fine-grained architecture-preserving pruning
ClickTrain: efficient and accurate end-to-end deep learning training via fine-grained architecture-preserving pruning
复制标题
DOI：
10.1145/3447818.3459988
复制
发表时间：
2020-11
期刊：
影响因子：
--
通讯作者：
Chengming Zhang;Geng Yuan;Wei Niu;Jiannan Tian;Sian Jin;Donglin Zhuang;Zhe Jiang;Yanzhi Wang;Bin Ren;S. Song;Dingwen Tao
中科院分区：
文献类型：
--
作者：
Chengming Zhang;Geng Yuan;Wei Niu;Jiannan Tian;Sian Jin;Donglin Zhuang;Zhe Jiang;Yanzhi Wang;Bin Ren;S. Song;Dingwen Tao
卷积神经网络（CNN）正变得越来越深，越来越宽，越来越非线性，因为对预测精度和分析质量的需求不断增长。然而，广泛而深入的CNN需要大量的计算资源和处理时间。许多先前的工作已经研究了模型修剪，以提高推理性能，但很少做的工作，有效地降低训练成本。在本文中，我们提出了ClickTrain：一个高效准确的CNN端到端训练和修剪框架。与现有的训练过程中的修剪工作不同，ClickTrain通过细粒度的结构保留修剪提供更高的模型精度和压缩比。通过利用基于模式的修剪与我们提出的新的准确权重重要性估计，动态模式生成和选择以及编译器辅助计算优化，ClickTrain生成高度准确和快速的修剪CNN模型，与基线训练相比，无需任何时间开销即可直接部署。ClickTrain还将最先进的训练后修剪方法的端到端时间成本降低了2.3倍，具有相当的准确性和压缩比。此外，与最先进的训练期间修剪方法相比，ClickTrain在类似的有限训练时间下，在测试的CNN模型和数据集上提供了显着的准确性和压缩比。
Convolutional neural networks (CNNs) are becoming increasingly deeper, wider, and non-linear because of the growing demand on prediction accuracy and analysis quality. The wide and deep CNNs, however, require a large amount of computing resources and processing time. Many previous works have studied model pruning to improve inference performance, but little work has been done for effectively reducing training cost. In this paper, we propose ClickTrain: an efficient and accurate end-to-end training and pruning framework for CNNs. Different from the existing pruning-during-training work, ClickTrain provides higher model accuracy and compression ratio via fine-grained architecture-preserving pruning. By leveraging pattern-based pruning with our proposed novel accurate weight importance estimation, dynamic pattern generation and selection, and compiler-assisted computation optimizations, ClickTrain generates highly accurate and fast pruned CNN models for direct deployment without any time overhead, compared with the baseline training. ClickTrain also reduces the end-to-end time cost of the state-of-the-art pruning-after-training method by up to 2.3x with comparable accuracy and compression ratio. Moreover, compared with the state-of-the-art pruning-during-training approach, ClickTrain provides significant improvements both accuracy and compression ratio on the tested CNN models and datasets, under similar limited training time.