Getting Away with More Network Pruning: From Sparsity to Geometry and Linear Regions
Getting Away with More Network Pruning: From Sparsity to Geometry and Linear Regions
复制标题
DOI：
10.48550/arxiv.2301.07966
复制
发表时间：
2023-01
期刊：
影响因子：
--
通讯作者：
Junyang Cai; Nguyen-Khai-Nguyen-Nguyen;Nishant Shrestha;Aidan Good;Ruisen Tu;Xin Yu;Shandian Zhe;Thiago Serra
中科院分区：
文献类型：
--
作者：
Junyang Cai; Nguyen-Khai-Nguyen-Nguyen;Nishant Shrestha;Aidan Good;Ruisen Tu;Xin Yu;Shandian Zhe;Thiago Serra
神经网络的一个令人惊讶的特点是，它们的连接可以在很大程度上被修剪，而对准确性几乎没有影响。但是，当我们越过参数稀疏的临界水平时，任何进一步的修剪都会导致准确性的突然下降。这种下降合理地反映了模型复杂性的损失，这是我们的目标。在这项工作中，我们探讨了稀疏性如何影响神经网络定义的线性区域的几何形状，从而减少了基于架构的线性区域的预期最大数量。我们观察到，修剪影响精度类似于稀疏性如何影响线性区域的数量和我们提出的最大数量的界限。相反，我们发现，与在所有层中使用相同稀疏度进行修剪相比，选择跨层的稀疏度以最大化我们的界限通常会提高准确性，从而为我们提供修剪位置的指导。
One surprising trait of neural networks is the extent to which their connections can be pruned with little to no effect on accuracy. But when we cross a critical level of parameter sparsity, pruning any further leads to a sudden drop in accuracy. This drop plausibly reflects a loss in model complexity, which we aim to avoid. In this work, we explore how sparsity also affects the geometry of the linear regions defined by a neural network, and consequently reduces the expected maximum number of linear regions based on the architecture. We observe that pruning affects accuracy similarly to how sparsity affects the number of linear regions and our proposed bound for the maximum number. Conversely, we find out that selecting the sparsity across layers to maximize our bound very often improves accuracy in comparison to pruning as much with the same sparsity in all layers, thereby providing us guidance on where to prune.