The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networks
The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networks
复制标题
DOI：
10.48550/arxiv.2203.04466
复制
发表时间：
2022-03
期刊：
影响因子：
--
通讯作者：
Xin Yu;Thiago Serra;Srikumar Ramalingam;Shandian Zhe
中科院分区：
文献类型：
--
作者：
Xin Yu;Thiago Serra;Srikumar Ramalingam;Shandian Zhe
如果神经网络的规模更大，它们往往会通过训练获得更好的准确性——即使最终的模型是过度参数化的。然而，在训练前、训练中或训练后仔细去除这些多余的参数，也可能产生精度相似甚至更高的模型。在许多情况下，这可以通过简单的启发式方法来实现，就像删除绝对值最小的权重百分比一样——尽管大小并不是权重相关性的完美代表。考虑到从剪枝中获得更好的性能取决于考虑去除多个权重的综合效果，我们重新审视了基于影响的剪枝的经典方法之一：最优脑外科医生（OBS）。我们提出了一种易于处理的启发式方法来解决OBS的组合扩展问题，其中我们选择同时删除的权重，以及系统地更新剩余的权重。我们的选择方法在高稀疏度下优于其他方法，并且权重更新即使与其他方法结合使用也具有优势。
Neural networks tend to achieve better accuracy with training if they are larger -- even if the resulting models are overparameterized. Nevertheless, carefully removing such excess parameters before, during, or after training may also produce models with similar or even improved accuracy. In many cases, that can be curiously achieved by heuristics as simple as removing a percentage of the weights with the smallest absolute value -- even though magnitude is not a perfect proxy for weight relevance. With the premise that obtaining significantly better performance from pruning depends on accounting for the combined effect of removing multiple weights, we revisit one of the classic approaches for impact-based pruning: the Optimal Brain Surgeon(OBS). We propose a tractable heuristic for solving the combinatorial extension of OBS, in which we select weights for simultaneous removal, as well as a systematic update of the remaining weights. Our selection method outperforms other methods under high sparsity, and the weight update is advantageous even when combined with the other methods.