DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedule-猫眼课题宝

DOI：

10.48550/arxiv.2302.12022

发表时间：

2023-02

期刊：

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Maor Ivgi;Oliver Hinder;Y. Carmon

我们提出了一个无调谐的动态SGD步长公式，我们称之为距离的连续性（DoG）。DoG步长取决于简单的经验量（距初始点的距离和梯度的范数），并且没有“学习率”参数。从理论上讲，我们表明，轻微的变化的DoG公式享有强参数无收敛保证随机凸优化假设只有\n {局部有界}随机梯度。从经验上讲，我们考虑了广泛的视觉和语言迁移学习任务，并表明，DoG的性能接近SGD的调整学习率。我们还提出了一个每层的变体的狗，一般优于调整SGD，接近性能的调整亚当。PyTorch实现可在https://github.com/formll/dog上获得

We propose a tuning-free dynamic SGD step size formula, which we call Distance over Gradients (DoG). The DoG step sizes depend on simple empirical quantities (distance from the initial point and norms of gradients) and have no ``learning rate'' parameter. Theoretically, we show that a slight variation of the DoG formula enjoys strong parameter-free convergence guarantees for stochastic convex optimization assuming only \emph{locally bounded} stochastic gradients. Empirically, we consider a broad range of vision and language transfer learning tasks, and show that DoG's performance is close to that of SGD with tuned learning rate. We also propose a per-layer variant of DoG that generally outperforms tuned SGD, approaching the performance of tuned Adam. A PyTorch implementation is available at https://github.com/formll/dog

DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedule

文献摘要

相似文献