Improving Deep Neural Networks’ Training for Image Classification With Nonlinear Conjugate Gradient-Style Adaptive Momentum
Improving Deep Neural Networks’ Training for Image Classification With Nonlinear Conjugate Gradient-Style Adaptive Momentum
复制标题
使用非线性共轭梯度式自适应动量改进深度神经网络 - 图像分类训练
DOI：
10.1109/tnnls.2023.3255783
复制
发表时间：
2023
影响因子：
10.4
通讯作者：
Ye, Qiang
中科院分区：
文献类型：
--
作者：
Wang, Bao;Ye, Qiang
在基于随机梯度的优化算法中，动量对于加速或改进深度神经网络 (DNN) 的训练至关重要。在深度学习实践中，动量通常由一个经过良好校准的常数来加权。然而，调整动量超参数可能会带来很大的计算负担。在本文中，我们提出了一种新的自适应动量来改进 DNN 训练；这种自适应动量不需要与动量相关的超参数，是由非线性共轭梯度（NCG）方法驱动的。具有这种新的自适应动量的随机梯度下降 (SGD) 消除了动量超参数校准的需要，允许使用显着更大的学习率，加速 DNN 训练，并提高训练的 DNN 的最终准确性和鲁棒性。例如，具有这种自适应动量的 SGD 将 CIFAR10 和 CIFAR100 的 ResNet110 训练的分类错误分别从 5.25% 减少到 4.64%，从 23.75% 减少到 20.03%。此外，SGD 具有新的自适应动量，也有利于对抗性训练，从而提高了训练后的 DNN 的对抗性鲁棒性。
Momentum is crucial in stochastic gradient-based optimization algorithms for accelerating or improving training deep neural networks (DNNs). In deep learning practice, the momentum is usually weighted by a well-calibrated constant. However, tuning the hyperparameter for momentum can be a significant computational burden. In this article, we propose a novel adaptive momentum for improving DNNs training; this adaptive momentum, with no momentum-related hyperparameter required, is motivated by the nonlinear conjugate gradient (NCG) method. Stochastic gradient descent (SGD) with this new adaptive momentum eliminates the need for the momentum hyperparameter calibration, allows using a significantly larger learning rate, accelerates DNN training, and improves the final accuracy and robustness of the trained DNNs. For instance, SGD with this adaptive momentum reduces classification errors for training ResNet110 for CIFAR10 and CIFAR100 from 5.25% to 4.64% and 23.75% to 20.03%, respectively. Furthermore, SGD, with the new adaptive momentum, also benefits adversarial training and, hence, improves the adversarial robustness of the trained DNNs.