A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performance
A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performance
复制标题
DOI：
--
复制
发表时间：
2020-02
期刊：
影响因子：
--
通讯作者：
Xiaoyun Li;Zhenxun Zhuang;Francesco Orabona
中科院分区：
文献类型：
--
作者：
Xiaoyun Li;Zhenxun Zhuang;Francesco Orabona
随机梯度下降法（SGD）是训练大规模机器学习模型的常用工具。然而，它的性能变化很大，关键取决于步长的选择。因此，人们提出了各种调整步长的策略，包括从坐标出发的方法（又称 "自适应 "步长）到在每次迭代中改变步长的复杂启发式方法。在本文中，我们研究了两种步长计划，它们的威力已在实践中得到反复证实：指数步长和余弦步长。我们首次为它们提供了理论支持，证明了光滑非凸函数的收敛率，无论是否存在 Polyak-\L{}ojasiewicz (PL) 条件。此外，我们还展示了一个令人惊讶的特性，即这两种策略对 PL 函数随机梯度中的噪声水平具有\emph{adaptive}适应性。也就是说，与多项式步长相反，它们不需要知道噪声水平，也不需要根据噪声水平调整超参数，就能达到几乎最优的性能。最后，我们利用深度学习架构对现实世界的数据集进行了公平、全面的实证评估。结果表明，即使只需要调整最多两个超参数，这两种策略也能达到最佳或与各种经过精细调整的最先进策略的性能相匹配。
Stochastic Gradient Descent (SGD) is a popular tool in training large-scale machine learning models. Its performance, however, is highly variable, depending crucially on the choice of the step sizes. Accordingly, a variety of strategies for tuning the step sizes have been proposed, ranging from coordinate-wise approaches (a.k.a. ``adaptive'' step sizes) to sophisticated heuristics to change the step size in each iteration. In this paper, we study two step size schedules whose power has been repeatedly confirmed in practice: the exponential and the cosine step sizes. For the first time, we provide theoretical support for them proving convergence rates for smooth non-convex functions, with and without the Polyak-\L{}ojasiewicz (PL) condition. Moreover, we show the surprising property that these two strategies are \emph{adaptive} to the noise level in the stochastic gradients of PL functions. That is, contrary to polynomial step sizes, they achieve almost optimal performance without needing to know the noise level nor tuning their hyperparameters based on it. Finally, we conduct a fair and comprehensive empirical evaluation of real-world datasets with deep learning architectures. Results show that, even if only requiring at most two hyperparameters to tune, these two strategies best or match the performance of various finely-tuned state-of-the-art strategies.