Gradient Descent: The Ultimate Optimizer
Gradient Descent: The Ultimate Optimizer
复制标题
DOI：
--
复制
发表时间：
2019-09
期刊：
影响因子：
--
通讯作者：
Kartik Chandra;Audrey Xie;Jonathan Ragan-Kelley;E. Meijer
中科院分区：
文献类型：
--
作者：
Kartik Chandra;Audrey Xie;Jonathan Ragan-Kelley;E. Meijer
使用任何基于梯度的机器学习算法的工作涉及调整优化器超参数（例如其步长）的繁琐任务。最近的工作表明，如何通过提前手动得出“高度级别”的表达式来与模型参数一起优化步长本身。我们展示了如何通过简单而优雅的重新传播来自动计算高升级物。这使我们能够轻松地将方法应用于其他优化器和超参数（例如动量系数）。我们甚至可以将该方法递归地应用于其自己的超级参赛者，等等。随着这些优化器的这些塔越高，它们对最初选择超参数的敏感也不太敏感。我们提出了对MLP，CNN和RNN验证此验证的实验。最后，我们提供了该算法的简单pytorch实现（请参阅People.csail.mit.edu/kach/gradient-descent-the-yultimate-optimizer）。
Working with any gradient-based machine learning algorithm involves the tedious task of tuning the optimizer's hyperparameters, such as its step size. Recent work has shown how the step size can itself be optimized alongside the model parameters by manually deriving expressions for"hypergradients"ahead of time. We show how to automatically compute hypergradients with a simple and elegant modification to backpropagation. This allows us to easily apply the method to other optimizers and hyperparameters (e.g. momentum coefficients). We can even recursively apply the method to its own hyper-hyperparameters, and so on ad infinitum. As these towers of optimizers grow taller, they become less sensitive to the initial choice of hyperparameters. We present experiments validating this for MLPs, CNNs, and RNNs. Finally, we provide a simple PyTorch implementation of this algorithm (see people.csail.mit.edu/kach/gradient-descent-the-ultimate-optimizer).