Maximal Initial Learning Rates in Deep ReLU Networks
Maximal Initial Learning Rates in Deep ReLU Networks
复制标题
DOI：
10.48550/arxiv.2212.07295
复制
发表时间：
2022-12
期刊：
影响因子：
--
通讯作者：
Gaurav M. Iyer;B. Hanin;D. Rolnick
中科院分区：
文献类型：
--
作者：
Gaurav M. Iyer;B. Hanin;D. Rolnick
训练神经网络需要选择一个合适的学习率，这涉及到收敛速度和收敛效果之间的权衡。虽然对于学习率可以有多大已经有了大量的理论和实证分析，但大多数先前的工作只关注后期训练。在这项工作中，我们引入了最大初始学习率$\eta^{\ast}$——一个随机初始化的神经网络能够成功开始训练并达到（至少）给定阈值精度的最大学习率。通过一种简单的方法来估计$\eta^{\ast}$，我们观察到在等宽全连接ReLU网络中，$\eta^{\ast}$在训练后期与最大学习率的表现不同。具体来说，我们发现如果（i）网络的宽度与深度相比足够大，并且（ii）输入层以相对较小的学习率进行训练，那么$\eta^{\ast}$可以很好地预测为深度×宽度的幂次方。我们进一步分析了$\eta^{\ast}$与网络初始化时的尖锐度$\lambda_{1}$之间的关系，表明它们密切相关但不是反比关系。我们根据深度×宽度正式证明了$\lambda_{1}$的界限，这与我们的实证结果相符。
Training a neural network requires choosing a suitable learning rate, which involves a trade-off between speed and effectiveness of convergence. While there has been considerable theoretical and empirical analysis of how large the learning rate can be, most prior work focuses only on late-stage training. In this work, we introduce the maximal initial learning rate $\eta^{\ast}$ - the largest learning rate at which a randomly initialized neural network can successfully begin training and achieve (at least) a given threshold accuracy. Using a simple approach to estimate $\eta^{\ast}$, we observe that in constant-width fully-connected ReLU networks, $\eta^{\ast}$ behaves differently from the maximum learning rate later in training. Specifically, we find that $\eta^{\ast}$ is well predicted as a power of depth $\times$ width, provided that (i) the width of the network is sufficiently large compared to the depth, and (ii) the input layer is trained at a relatively small learning rate. We further analyze the relationship between $\eta^{\ast}$ and the sharpness $\lambda_{1}$ of the network at initialization, indicating they are closely though not inversely related. We formally prove bounds for $\lambda_{1}$ in terms of depth $\times$ width that align with our empirical results.