A One-Shot Reparameterization Method for Reducing the Loss of Tile Pruning on DNNs
A One-Shot Reparameterization Method for Reducing the Loss of Tile Pruning on DNNs
复制标题
DOI：
10.1109/ijcnn55064.2022.9889789
复制
发表时间：
2022-07
期刊：
影响因子：
--
通讯作者：
Yancheng Li;Qingzhong Ai;Fumihiko Ino
中科院分区：
文献类型：
--
作者：
Yancheng Li;Qingzhong Ai;Fumihiko Ino
最近，瓦片修剪被广泛研究以加速深度神经网络（DNN）的推理。然而，我们发现，由于瓦片修剪，可以消除重要元素和不重要元素，在训练的DNN上损失很大。在这项研究中，我们提出了一个一次性的重新参数化方法，称为TileTrans，以减少瓷砖修剪的损失。具体来说，我们重新配置权重矩阵的行或列，使得模型架构在重新参数化后保持不变。这种重新排列实现了DNN模型的重新参数化，而无需任何重新训练。所提出的重新参数化方法将重要元素组合到同一个瓦片中，从而在瓦片修剪后保留重要元素。此外，TileTrans可以无缝集成到现有的瓦片修剪方法中，因为它是在修剪之前执行的预处理方法，这与大多数现有方法正交。实验结果表明，我们的方法是必不可少的减少损失的瓷砖修剪DNN。具体来说，AlexNet的准确性提高了17%，而ResNet-34的准确性提高了5%，其中两个模型都在ImageNet上进行了预训练。
Recently, tile pruning has been widely studied to accelerate the inference of deep neural networks (DNNs). However, we found that the loss due to tile pruning, which can eliminate important elements together with unimportant elements, is large on trained DNNs. In this study, we propose a one-shot reparameterization method, called TileTrans, to reduce the loss of tile pruning. Specifically, we repermute the rows or columns of the weight matrix such that the model architecture can be kept unchanged after reparameterization. This repermutation realizes the reparameterization of the DNN model without any retraining. The proposed reparameterization method combines important elements into the same tile; thus, preserving the important elements after the tile pruning. Furthermore, TileTrans can be seamlessly integrated into existing tile pruning methods because it is a pre-processing method executed before pruning, which is orthogonal to most existing methods. The experimental results demonstrate that our method is essential in reducing the loss of tile pruning on DNNs. Specifically, the accuracy is improved by up to 17% for AlexNet while 5% for ResNet-34, where both models are pre-trained on ImageNet.