Transformed $\ell_1$ Regularization for Learning Sparse Deep Neural Networks
Transformed $\ell_1$ Regularization for Learning Sparse Deep Neural Networks
复制标题
DOI：
--
复制
发表时间：
2019-01
期刊：
影响因子：
--
通讯作者：
Rongrong Ma;Jianyu Miao;Lingfeng Niu;Peng Zhang
中科院分区：
文献类型：
--
作者：
Rongrong Ma;Jianyu Miao;Lingfeng Niu;Peng Zhang
深度神经网络（DNN）在许多领域取得了非凡的成功。然而，为了实现这一成功，DNN通常携带大量的权重参数，导致内存和计算资源的巨大成本。当训练数据不足时，这种网络也可能发生过拟合。这些缺点严重阻碍了DNN在资源受限平台中的应用。事实上，许多网络权重都是冗余的，可以从网络中删除，而不会对性能造成太大损失。为此，我们引入了一个新的非凸集成变换$\ell_1 $正则化，以促进DNN的稀疏性，同时删除冗余连接和不必要的神经元。具体地说，我们将变换后的$\ell_1$应用到网络权值的矩阵空间中，并利用它来去除冗余连接。此外，群体稀疏性也被用来作为一个辅助，以消除不必要的神经元。同时提出了一种有效的随机邻近梯度算法来求解新模型。据我们所知，这是第一个在基于稀疏优化的方法中利用非凸正则化器来提高DNN稀疏性的工作。在多个公开数据集上的实验证明了该方法的有效性。
Deep neural networks (DNNs) have achieved extraordinary success in numerous areas. However, to attain this success, DNNs often carry a large number of weight parameters, leading to heavy costs of memory and computation resources. Overfitting is also likely to happen in such network when the training data are insufficient. These shortcomings severely hinder the application of DNNs in resource-constrained platforms. In fact, many network weights are known to be redundant and can be removed from the network without much loss of performance. To this end, we introduce a new non-convex integrated transformed $\ell_1$ regularizer to promote sparsity for DNNs, which removes both redundant connections and unnecessary neurons simultaneously. To be specific, we apply the transformed $\ell_1$ to the matrix space of network weights and utilize it to remove redundant connections. Besides, group sparsity is also employed as an auxiliary to remove unnecessary neurons. An efficient stochastic proximal gradient algorithm is presented to solve the new model at the same time. To the best of our knowledge, this is the first work to utilize a non-convex regularizer in sparse optimization based method to promote sparsity for DNNs. Experiments on several public datasets demonstrate the effectiveness of the proposed method.