Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability
Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability
复制标题
DOI：
10.48550/arxiv.2209.15594
复制
发表时间：
2022-09
期刊：
影响因子：
--
通讯作者：
Alexandru Damian;Eshaan Nichani;Jason D. Lee
中科院分区：
文献类型：
--
作者：
Alexandru Damian;Eshaan Nichani;Jason D. Lee
传统的梯度下降分析表明，当 Hessian 矩阵的最大特征值（也称为锐度 $S(\theta)$）以 $2/\eta$ 为界时，训练是“稳定”的，训练损失单调递减。然而，最近的研究发现，在使用全批量或大批量梯度下降训练现代神经网络时，这种假设并不成立。最近，科恩等人。 （2021）观察到两个重要现象。第一个被称为渐进锐化，锐度在整个训练过程中稳步增加，直到达到不稳定截止值 $2/\eta$。第二个被称为稳定性边缘的是，在剩余的训练中，锐度徘徊在 $2/\eta$，而损失继续下降，尽管是非单调的。我们证明，稳定边缘的梯度下降动态远非混乱，可以通过三次泰勒展开来捕获：当迭代由于不稳定而在 Hessian 顶部特征向量的方向上发散时，损失函数的局部泰勒展开中的三次项会导致曲率减小，直到恢复稳定性。这种性质，我们称之为自稳定，是梯度下降的一般性质，解释了它在稳定边缘的行为。自稳定的一个关键结果是，在$S(\theta)\le 2/\eta$约束下，稳定边缘的梯度下降隐式遵循投影梯度下降（PGD）。我们的分析提供了对整个训练过程中 PGD 轨迹的损失、锐度和偏差的精确预测，我们在许多标准设置中进行了经验验证，并在温和条件下进行了理论上验证。我们的分析揭示了梯度下降隐含的稳定性偏差机制。
Traditional analyses of gradient descent show that when the largest eigenvalue of the Hessian, also known as the sharpness $S(\theta)$, is bounded by $2/\eta$, training is"stable"and the training loss decreases monotonically. Recent works, however, have observed that this assumption does not hold when training modern neural networks with full batch or large batch gradient descent. Most recently, Cohen et al. (2021) observed two important phenomena. The first, dubbed progressive sharpening, is that the sharpness steadily increases throughout training until it reaches the instability cutoff $2/\eta$. The second, dubbed edge of stability, is that the sharpness hovers at $2/\eta$ for the remainder of training while the loss continues decreasing, albeit non-monotonically. We demonstrate that, far from being chaotic, the dynamics of gradient descent at the edge of stability can be captured by a cubic Taylor expansion: as the iterates diverge in direction of the top eigenvector of the Hessian due to instability, the cubic term in the local Taylor expansion of the loss function causes the curvature to decrease until stability is restored. This property, which we call self-stabilization, is a general property of gradient descent and explains its behavior at the edge of stability. A key consequence of self-stabilization is that gradient descent at the edge of stability implicitly follows projected gradient descent (PGD) under the constraint $S(\theta) \le 2/\eta$. Our analysis provides precise predictions for the loss, sharpness, and deviation from the PGD trajectory throughout training, which we verify both empirically in a number of standard settings and theoretically under mild conditions. Our analysis uncovers the mechanism for gradient descent's implicit bias towards stability.