The Power of Preconditioning in Overparameterized Low-Rank Matrix Sensing
The Power of Preconditioning in Overparameterized Low-Rank Matrix Sensing
复制标题
DOI：
10.48550/arxiv.2302.01186
复制
发表时间：
2023-02
期刊：
影响因子：
--
通讯作者：
Xingyu Xu;Yandi Shen;Yuejie Chi;Cong Ma
中科院分区：
文献类型：
--
作者：
Xingyu Xu;Yandi Shen;Yuejie Chi;Cong Ma
我们建议$ \ textsf {scaledgd（$ \ lambda $）} $，这是一种预处理的梯度下降方法，用于解决最低级别等级时解决低率矩阵传感问题，而当矩阵可能不适合矩阵时。使用过度隔离的因子表示，$ \ textsf {scaledgd（$ \ lambda $）} $从一个小的随机初始化开始，并以梯度下降进行以特定形式的阻尼预处理，以对抗由过度参数化和不良条件诱导的不良曲线。以预先调节器产生的轻度计算间接开销，$ \ textsf {scaledgd（$ \ \ lambda $）} $即使在过量的情况下，与香草梯度下降（$ \ textsf {gd} $）相比，对不良调节非常强大。具体而言，我们表明，在高斯设计下，$ \ textsf {scaledgd（$ \ \ lambda $）} $收敛于真实的低率矩阵，以少量的迭代量后，以恒定的线性速率收敛于少量的线性速率，该迭代仅在与对数时相对于对数的缩放。条件编号和问题维度。这显着提高了vextsf {gd} $的收敛速率，该{gd} $受到多项式依赖条件编号的依赖。我们的工作提供了有关加速融合的预处理的证据，而不会损害过度参数化学习中的概括。
We propose $\textsf{ScaledGD($\lambda$)}$, a preconditioned gradient descent method to tackle the low-rank matrix sensing problem when the true rank is unknown, and when the matrix is possibly ill-conditioned. Using overparametrized factor representations, $\textsf{ScaledGD($\lambda$)}$ starts from a small random initialization, and proceeds by gradient descent with a specific form of damped preconditioning to combat bad curvatures induced by overparameterization and ill-conditioning. At the expense of light computational overhead incurred by preconditioners, $\textsf{ScaledGD($\lambda$)}$ is remarkably robust to ill-conditioning compared to vanilla gradient descent ($\textsf{GD}$) even with overprameterization. Specifically, we show that, under the Gaussian design, $\textsf{ScaledGD($\lambda$)}$ converges to the true low-rank matrix at a constant linear rate after a small number of iterations that scales only logarithmically with respect to the condition number and the problem dimension. This significantly improves over the convergence rate of vanilla $\textsf{GD}$ which suffers from a polynomial dependency on the condition number. Our work provides evidence on the power of preconditioning in accelerating the convergence without hurting generalization in overparameterized learning.