Greedy Optimization Provably Wins the Lottery: Logarithmic Number of Winning Tickets is Enough
Greedy Optimization Provably Wins the Lottery: Logarithmic Number of Winning Tickets is Enough
复制标题
DOI：
--
复制
发表时间：
2020-10
期刊：
影响因子：
--
通讯作者：
Mao Ye;Lemeng Wu;Qiang Liu
中科院分区：
文献类型：
--
作者：
Mao Ye;Lemeng Wu;Qiang Liu
尽管深度学习取得了巨大成功，但最近的研究表明，大型深度神经网络通常具有高度冗余性，并且可以显着减小尺寸。然而，在给定指定的精度下降容限的情况下，我们可以对神经网络进行多少修剪的理论问题仍然悬而未决。本文通过提出一种基于贪婪优化的剪枝方法为这个问题提供了一个答案。所提出的方法保证了修剪后的网络和原始网络之间的差异以指数快速速率衰减。在适用于大多数实际设置的弱假设下，修剪网络的大小。根据经验，我们的方法改进了修剪各种网络架构（包括 ImageNet 上的 ResNet、MobilenetV2/V3）的现有技术。
Despite the great success of deep learning, recent works show that large deep neural networks are often highly redundant and can be significantly reduced in size. However, the theoretical question of how much we can prune a neural network given a specified tolerance of accuracy drop is still open. This paper provides one answer to this question by proposing a greedy optimization based pruning method. The proposed method has the guarantee that the discrepancy between the pruned network and the original network decays with exponentially fast rate w.r.t. the size of the pruned network, under weak assumptions that apply for most practical settings. Empirically, our method improves prior arts on pruning various network architectures including ResNet, MobilenetV2/V3 on ImageNet.