Network Pruning via Transformable Architecture Search
Network Pruning via Transformable Architecture Search
复制标题
DOI：
--
复制
发表时间：
2019-05
期刊：
影响因子：
--
通讯作者：
Xuanyi Dong;Yi Yang
中科院分区：
文献类型：
--
作者：
Xuanyi Dong;Yi Yang
网络修剪降低了过参数化网络的计算成本，而不会损害性能。现有的修剪算法预先定义修剪后网络的宽度和深度，然后将参数从未修剪的网络传递到修剪后的网络。为了打破修剪网络的结构限制，我们提出应用神经结构搜索直接搜索具有灵活通道和层大小的网络。通过最小化修剪网络的损失来学习通道/层的数量。修剪后的网络的特征图是K个特征图片段（由K个不同大小的网络生成）的聚合，这些特征图片段基于概率分布进行采样。损失不仅可以反向传播到网络权重，还可以反向传播到参数化分布，以显式调整通道/层的大小。具体来说，我们应用通道插值来保持具有不同通道大小的特征图在聚合过程中对齐。每个分布中的大小的最大概率用作修剪网络的宽度和深度，其参数通过知识转移来学习，例如，从原始网络中提炼知识。在CIFAR-10、CIFAR-100和ImageNet上的实验结果表明，与传统的网络剪枝算法相比，本文提出的网络剪枝新方法是有效的。各种搜索和知识转移的方法进行了显示两个组件的有效性。代码在：这个https URL。
Network pruning reduces the computation costs of an over-parameterized network without performance damage. Prevailing pruning algorithms pre-define the width and depth of the pruned networks, and then transfer parameters from the unpruned network to pruned networks. To break the structure limitation of the pruned networks, we propose to apply neural architecture search to search directly for a network with flexible channel and layer sizes. The number of the channels/layers is learned by minimizing the loss of the pruned networks. The feature map of the pruned network is an aggregation of K feature map fragments (generated by K networks of different sizes), which are sampled based on the probability distribution.The loss can be back-propagated not only to the network weights, but also to the parameterized distribution to explicitly tune the size of the channels/layers. Specifically, we apply channel-wise interpolation to keep the feature map with different channel sizes aligned in the aggregation procedure. The maximum probability for the size in each distribution serves as the width and depth of the pruned network, whose parameters are learned by knowledge transfer, e.g., knowledge distillation, from the original networks. Experiments on CIFAR-10, CIFAR-100 and ImageNet demonstrate the effectiveness of our new perspective of network pruning compared to traditional network pruning algorithms. Various searching and knowledge transfer approaches are conducted to show the effectiveness of the two components. Code is at: this https URL.