Fast Learning of Deep Neural Networks via Singular Value Decomposition
Fast Learning of Deep Neural Networks via Singular Value Decomposition
复制标题
DOI：
10.1007/978-3-319-13560-1_65
复制
发表时间：
2014-12
期刊：
影响因子：
--
通讯作者：
Chenghao Cai;Dengfeng Ke;Yanyan Xu;Kaile Su
中科院分区：
文献类型：
--
作者：
Chenghao Cai;Dengfeng Ke;Yanyan Xu;Kaile Su
本文提出了一种基于奇异值分解（SVD）的深度神经网络快速学习方法。快速训练方法采用监督预调整过程，对dnn的权值矩阵参数进行粗略调整，改变奇异值的分布。将奇异值分解应用于预调整的深度神经网络，减少了深度神经网络中的参数数量。采用一种非常规的BP （Back Propagation）算法对SVD重构后的模型进行训练，该算法具有比传统BP算法更低的时间复杂度。实验结果表明，在大词汇量连续语音识别（LVCSR）任务上，采用快速训练方法，非常规BP算法在不损失识别性能的情况下实现了近2倍的提速，在不损失识别性能的情况下实现了近4倍的提速。
In this paper, we propose a new fast training methodology for learning of Deep Neural Networks (DNNs) via Singular Value Decomposition (SVD). The fast training methodology uses a supervised pre-adjusting process to adjust roughly parameters of weight matrices of DNNs and change distributions of singular values. SVD is applied to pre-adjusted DNNs, reducing quantities of parameters in DNNs. An unconventional Back Propagation (BP) algorithm is used to train the models restructured by SVD, which has lower time complexity than the conventional BP algorithm. Experimental results indicate that on Large Vocabulary Continuous Speech Recognition (LVCSR) tasks, using the fast training methodology, the unconventional BP algorithm achieves almost 2 times speed-up without any loss of recognition performance and almost 4 times speed-up with only a tiny loss of recognition performance.