A deterministic gradient-based approach to avoid saddle points
A deterministic gradient-based approach to avoid saddle points
复制标题
一种避免鞍点的基于确定性梯度的方法
DOI：
10.1017/s0956792522000316
复制
发表时间：
2022
影响因子：
1.9
通讯作者：
Wang, B.
中科院分区：
文献类型：
--
作者：
Kreusser, L. M.;Osher, S. J.;Wang, B.
具有大量鞍点的损失函数是有效训练现代机器学习模型的主要障碍之一。一阶方法，如梯度下降(GD)，通常是训练ML模型的首选方法。然而，对于某些初始猜测的选择，这些方法收敛到鞍点。本文对最近提出的拉普拉斯平滑梯度下降算法[Osher et al.，arxiv：1806.06317]提出了一种改进的改进算法，称为改进的拉普拉斯平滑梯度下降算法(MLSGD)，并证明了它在不牺牲收敛速度的前提下避免了鞍点的潜力。我们的分析是基于吸引区，该吸引区由所考虑的数值格式收敛到鞍点的所有起点形成。我们从解析和数值两个方面研究了吸引区的维度。对于一类标准的二次函数，我们证明了mLSGD的吸引域的维度为。
Loss functions with a large number of saddle points are one of the major obstacles for training modern machine learning (ML) models efficiently. First-order methods such as gradient descent (GD) are usually the methods of choice for training ML models. However, these methods converge to saddle points for certain choices of initial guesses. In this paper, we propose a modification of the recently proposed Laplacian smoothing gradient descent (LSGD) [Osher et al., arXiv:1806.06317], called modified LSGD (mLSGD), and demonstrate its potential to avoid saddle points without sacrificing the convergence rate. Our analysis is based on the attraction region, formed by all starting points for which the considered numerical scheme converges to a saddle point. We investigate the attraction region’s dimension both analytically and numerically. For a canonical class of quadratic functions, we show that the dimension of the attraction region for mLSGD is .