Win The Lottery Ticket Via Fourier Analysis: Frequencies Guided Network Pruning
Win The Lottery Ticket Via Fourier Analysis: Frequencies Guided Network Pruning
复制标题
DOI：
10.1109/icassp43922.2022.9746892
复制
发表时间：
2022-01
期刊：
影响因子：
--
通讯作者：
Yuzhang Shang;Bin Duan;Ziliang Zong;Liqiang Nie;Yan Yan-Yan
中科院分区：
文献类型：
--
作者：
Yuzhang Shang;Bin Duan;Ziliang Zong;Liqiang Nie;Yan Yan-Yan
随着深度学习的巨大成功，迫切需要高效的网络压缩算法来释放边缘设备（如智能手机或平板电脑）的潜在计算能力。然而，最优网络修剪是一个非平凡的任务，在数学上是一个NP难问题。以前的研究人员将训练修剪后的网络解释为购买彩票。在本文中，我们研究了基于幅度的剪枝（MBP）方案，并通过对深度学习模型的傅立叶分析从一个新的角度对其进行分析，以指导模型设计。除了使用傅立叶变换解释MBP的泛化能力外，我们还提出了一种新的两阶段修剪方法，其中一个阶段是获得修剪后网络的拓扑结构，另一个阶段是使用知识蒸馏在频域上从低到高重新训练修剪后的网络以恢复能力。在CIFAR-10和CIFAR-100上的大量实验表明，与其他传统的MBP算法相比，我们的新的基于傅立叶分析的MBP的优越性。
With the remarkable success of deep learning recently, efficient network compression algorithms are urgently demanded for releasing the potential computational power of edge devices, such as smartphones or tablets. However, optimal network pruning is a non-trivial task which mathematically is an NP-hard problem. Previous researchers explain training a pruned network as buying a lottery ticket. In this paper, we investigate the Magnitude-Based Pruning (MBP) scheme and analyze it from a novel perspective through Fourier analysis on the deep learning model to guide model designation. Besides explaining the generalization ability of MBP using Fourier transform, we also propose a novel two-stage pruning approach, where one stage is to obtain the topological structure of the pruned network and the other stage is to retrain the pruned network to recover the capacity using knowledge distillation from lower to higher on the frequency domain. Extensive experiments on CIFAR-10 and CIFAR-100 demonstrate the superiority of our novel Fourier analysis based MBP compared to other traditional MBP algorithms.