Fast DNN Training Based on Auxiliary Function Technique

Fast DNN Training Based on Auxiliary Function Technique
复制标题

基于辅助函数技术的快速DNN训练

DOI：
10.1109/icassp.2015.7178353
复制
发表时间：
2015
期刊：
Proc. ICASSP
影响因子：
--
通讯作者：
Nobutaka Ono and Emmanuel Vincent
Nobutaka Ono and Emmanuel Vincent
中科院分区：
--
文献类型：
--
作者：
Dung Tran;Nobutaka Ono and Emmanuel Vincent

文献摘要

相似文献

深度神经网络（DNN）通常使用固定学习率或自适应学习率方法（ADAGRAD）通过随机梯度下降（SGD）进行优化。在本文中，我们介绍了一种新的学习规则的神经网络，是基于辅助函数技术，无需参数调整。不是最小化目标函数，而是逐层递归引入二次辅助函数，该函数具有封闭形式的最优解。我们证明了新的学习规则的单调下降。实验结果表明，该算法比SGD算法收敛速度更快，收敛到更好的局部极小值。此外，我们提出了一个建议的学习规则和ADAGRAD，进一步加快收敛的组合。MNIST数据库上的实验评估表明，所提出的方法在数字识别精度方面的好处。
Deep neural networks (DNN) are typically optimized with stochastic gradient descent (SGD) using a fixed learning rate or an adaptive learning rate approach (ADAGRAD). In this paper, we introduce a new learning rule for neural networks that is based on an auxiliary function technique without parameter tuning. Instead of minimizing the objective function, a quadratic auxiliary function is recursively introduced layer by layer which has a closed-form optimum. We prove the monotonic decrease of the new learning rule. Our experiments show that the proposed algorithm converges faster and to a better local minimum than SGD. In addition, we propose a combination of the proposed learning rule and ADAGRAD which further accelerates convergence. Experimental evaluation on the MNIST database shows the benefit of the proposed approach in terms of digit recognition accuracy.