Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification
Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification
复制标题
DOI：
10.1109/cvprw50498.2020.00347
复制
发表时间：
2020-04
期刊：
影响因子：
--
通讯作者：
Huanrui Yang;Minxue Tang;W. Wen;Feng Yan;Daniel Hu;Ang Li;H. Li;Yiran Chen
中科院分区：
文献类型：
--
作者：
Huanrui Yang;Minxue Tang;W. Wen;Feng Yan;Daniel Hu;Ang Li;H. Li;Yiran Chen
现代深度神经网络（dnn）通常需要高内存消耗和大计算负载。为了在边缘或移动设备上有效地部署深度神经网络算法，研究人员探索了一系列深度神经网络压缩算法，包括分解方法。因式分解方法是用两个或多个低秩矩阵的乘法来近似DNN层的权重矩阵。然而，在训练过程中很难测量DNN层的等级。以往的工作主要是通过隐式逼近或在每个训练步骤中使用昂贵的奇异值分解（SVD）过程来诱导低秩。前者精度损失大，后者效率低。在这项工作中，我们提出了SVD训练，这是第一种在训练过程中明确实现低秩dnn的方法，而无需在每一步都应用SVD。SVD训练首先将每一层分解为其全秩SVD的形式，然后直接对分解的权值进行训练。在奇异向量上加入正交正则化，保证了奇异向量分解的有效形式，避免了梯度消失/爆炸。通过在每层的奇异值上应用稀疏性诱导正则化器来鼓励低秩。最后应用奇异值剪枝，显式地得到一个低秩模型。我们的经验表明，与之前的分解方法和目前最先进的滤波剪枝方法相比，SVD训练可以显著降低DNN层的秩，在相同精度下实现更高的计算量减少。
Modern deep neural networks (DNNs) often require high memory consumption and large computational loads. In order to deploy DNN algorithms efficiently on edge or mobile devices, a series of DNN compression algorithms have been explored, including factorization methods. Factorization methods approximate the weight matrix of a DNN layer with the multiplication of two or multiple low-rank matrices. However, it is hard to measure the ranks of DNN layers during the training process. Previous works mainly induce low-rank through implicit approximations or via costly singular value decomposition (SVD) process on every training step. The former approach usually induces a high accuracy loss while the latter has a low efficiency. In this work, we propose SVD training, the first method to explicitly achieve low-rank DNNs during training without applying SVD on every step. SVD training first decomposes each layer into the form of its full-rank SVD, then performs training directly on the decomposed weights. We add orthogonality regularization to the singular vectors, which ensure the valid form of SVD and avoid gradient vanishing/exploding. Low-rank is encouraged by applying sparsity-inducing regularizers on the singular values of each layer. Singular value pruning is applied at the end to explicitly reach a low-rank model. We empirically show that SVD training can significantly reduce the rank of DNN layers and achieve higher reduction on computation load under the same accuracy, comparing to not only previous factorization methods but also state-of-the-art filter pruning methods.