iSparse: Output Informed Sparsification of Neural Network
iSparse: Output Informed Sparsification of Neural Network
复制标题
DOI：
10.1145/3372278.3390688
复制
发表时间：
2020-06
期刊：
影响因子：
--
通讯作者：
Yash Garg;K. Candan
中科院分区：
文献类型：
--
作者：
Yash Garg;K. Candan
深度神经网络在各种多媒体应用中取得了前所未有的成功。然而，创建的网络通常非常复杂，具有大量可训练的边缘，需要大量的计算资源。我们注意到，许多成功的网络通常包含大量的冗余边。此外，许多这些边对整体网络性能的贡献可以忽略不计。在本文中，我们提出了一种新的iSparse框架，并通过实验证明，我们可以在不影响网络性能的情况下对网络进行稀疏化。iSparse利用一种新的边缘显著性得分E来确定边缘相对于最终网络输出的重要性。此外，iSparse既可以在训练模型时应用，也可以在预训练模型的基础上应用，使其成为一种无需再训练的方法，从而减少了计算开销。iSparse与Dropout、L1、DropConnect、Retraining-Free和Lottery-Ticket假说在基准数据集上的比较表明，iSparse导致了有效的网络稀疏化。
Deep neural networks have demonstrated unprecedented success in various multimedia applications. However, the networks created are often very complex, with large numbers of trainable edges that require extensive computational resources. We note that many successful networks nevertheless often contain large numbers of redundant edges. Moreover, many of these edges may have negligible contributions towards the overall network performance. In this paper, we propose a novel iSparse framework and experimentally show, that we can sparsify the network without impacting the network performance. iSparse leverages a novel edge significance score, E, to determine the importance of an edge with respect to the final network output. Furthermore, iSparse can be applied both while training a model or on top of a pre-trained model, making it a retraining-free approach - leading to a minimal computational overhead. Comparisons of iSparse against Dropout, L1, DropConnect, Retraining-Free, and Lottery-Ticket Hypothesis on benchmark datasets show that iSparse leads to effective network sparsifications.