Local Regularizer Improves Generalization
Local Regularizer Improves Generalization
复制标题
DOI：
10.1609/aaai.v34i04.6167
复制
发表时间：
2020-04
期刊：
影响因子：
--
通讯作者：
Yikai Zhang;Hui Qu;Dimitris N. Metaxas;Chao Chen
中科院分区：
文献类型：
--
作者：
Yikai Zhang;Hui Qu;Dimitris N. Metaxas;Chao Chen
正则化在深度学习的泛化中起着重要的作用。在本文中，我们研究了深度学习中训练算法的无偏正则器的泛化能力。我们专注于局部正则化随机梯度下降（LRSGD）的训练方法。LRSGD利用梯度下降步骤中的近端类型惩罚来正则化训练中的SGD。研究表明，通过仔细选择相关参数，LRSGD的泛化效果优于SGD。我们彻底的理论分析得到了实验证据的支持。它推进了我们对深度学习的理论理解，并为设计训练算法提供了新的视角。代码可在https://github.com/huiqu18/LRSGD上获得。
Regularization plays an important role in generalization of deep learning. In this paper, we study the generalization power of an unbiased regularizor for training algorithms in deep learning. We focus on training methods called Locally Regularized Stochastic Gradient Descent (LRSGD). An LRSGD leverages a proximal type penalty in gradient descent steps to regularize SGD in training. We show that by carefully choosing relevant parameters, LRSGD generalizes better than SGD. Our thorough theoretical analysis is supported by experimental evidence. It advances our theoretical understanding of deep learning and provides new perspectives on designing training algorithms. The code is available at https://github.com/huiqu18/LRSGD.