Understanding the Dynamics of Gradient Flow in Overparameterized Linear models
Understanding the Dynamics of Gradient Flow in Overparameterized Linear models
复制标题
了解超参数化线性模型中梯度流的动力学
DOI：
10.48550/arxiv.2404.04454
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
René Vidal
中科院分区：
文献类型：
--
作者：
Salma Tarmoun;G. França;B. Haeffele;René Vidal
详细分析了超参数两层线性模型中梯度流的动力学行为。这个模型的一个特别有趣的特点是，由于大量的守恒定律约束系统遵循特定的轨迹，它的非线性动力学可以精确地求解。更准确地说，梯度流保持了输入和输出权重的Gramian矩阵的差异，其收敛到平衡既取决于差异的大小(在初始化时是固定的)，也取决于数据的频谱。另外，推广了以前的工作，我们证明了我们的结果，而不假设权重是小的、平衡的或谱的初始化。此外，我们在矩阵分解问题和Riccati类型的微分方程之间建立了有趣的数学联系。
We provide a detailed analysis of the dynamics of the gradient flow in overparameterized two-layer linear models. A particularly interesting feature of this model is that its nonlinear dynamics can be exactly solved as a consequence of a large number of conservation laws that constrain the system to follow particular trajectories. More precisely, the gradient flow preserves the difference of the Gramian matrices of the input and output weights, and its convergence to equilibrium depends on both the magnitude of that difference (which is fixed at initialization) and the spectrum of the data. In addition, and generalizing prior work, we prove our results without assuming small, balanced or spectral initialization for the weights. Moreover, we establish interesting mathematical connections between matrix factorization problems and differential equations of the Riccati type.
DOI：
10.1109/ita.2018.8503198
发表时间：
2017-05
期刊：
2018 Information Theory and Applications Workshop (ITA)
影响因子：
--
作者：
Suriya Gunasekar;Blake E. Woodworth;Srinadh Bhojanapalli;Behnam Neyshabur;N. Srebro
通讯作者：
Suriya Gunasekar;Blake E. Woodworth;Srinadh Bhojanapalli;Behnam Neyshabur;N. Srebro