Hyper-fast hyper-parameter tuning for the next generation of machine learning

机器学习（ML）使用大量数据来设置模型的参数，并且在越来越多的应用中取得了成功，从语音识别到计算机视觉再到语言翻译。但是机器学习有一个“超参数”的问题，即影响学习算法和模型结构的变量。为了获得良好的性能，我们通常仍然需要花费大量的时间来调整超参数。最近有人强调，调整当前ML语言模型的成本相当于五辆汽车在其整个生命周期中的碳排放量。这种情况只会变得更糟，因为下一代模型将拥有更多的超参数。在解决这个问题之前，我们将无法解决许多重要问题。该研究项目的长期目标是开发能够在短时间内在大量数据集上训练ML模型的算法。在过去的6年里，我们一直专注于开发更快收敛的算法，但我们现在需要转向处理超参数的紧迫问题。这个项目的短期目标是专注于解决与两个典型的超参数来源相关的问题：(A)用于学习的算法通常具有超参数，例如学习率。(B)正在使用的模型通常具有超参数，例如深度学习模型的深度。我们已经在(A)方面取得了进展。在2019年，我们给出了第一种方法，在训练过程中自动调整最重要的超参数之一——学习率。对于现代的“过度参数化”模型，该方法保证至少与最佳固定学习率一样好。我们计划开发对其他学习超参数不敏感的算法，并且不需要过度参数化假设。我的实验室在解决问题(B)方面具有独特的优势。当前调整网络架构参数的策略倾向于使用超参数的离散参数化。在10年前的工作中，我展示了多种使用连续参数化的方法，为涉及搜索图和超图结构的学习问题提供高质量的近似解决方案。这些连续松弛比以前基于离散参数化的方法提供了巨大的加速，我们将开发类似的新方法来解决调优模型超参数问题。拟议中的研究有巨大的潜在影响，潜在的应用范围从医学到科学发现再到自动驾驶汽车。在机器学习中，我们希望构建跨许多应用程序工作的算法和模型（我们希望“构建一个更好的锤子”，可以用于许多任务）。因此，机器学习模型底层算法的突破会立即影响到许多使用机器学习（或将在未来使用机器学习）的应用程序。

Machine learning (ML) uses large amounts data to set the parameters of a model, and is having success in a growing number of applications from speech recognition to computer vision to language translation. But ML has a problem with "hyper-parameters", the variables affecting the learning algorithm and the structure of the model. We typically still need to spend enormous amounts of time tuning hyper-parameters in order to obtain good performance. It was recently highlighted that the cost of tuning current ML language models has a comparable carbon output to five cars throughout their lifetime. This situation will only get worse as the next generation of models will have far more hyper-parameters. We will not be able to solve many important problems until we address this issue. The long-term goal of this research program is to develop algorithms that can train ML models on enormous datasets in a short amount of time. During the last 6 years we have focused on developing algorithms that converge faster, but we now need to turn to the pressing issue of dealing with the hyper-parameters. The short-term goal of this project is to focus on addressing the issues associated with the two typical sources of hyper-parameters: (A) The algorithm used to do the learning typically has hyper-parameters, such as the learning rate. (B) The model that is being used typically has hyper-parameters, such as the depth of a deep learning model. We have already made progress on (A). In 2019 we gave the first method that automatically tunes one of the most important hyper-parameters, the learning rate, during training. This method is guaranteed to perform at least as well as the best fixed learning rate for modern "over-parameterized" models. We plan to develop algorithms that are insensitive to other learning hyper-parameters and that do not require the over-parameterized assumption. My lab is uniquely positioned to address problem (B). Current strategies for tuning the parameters of network architectures tend to use discrete parameterizations of the hyper-parameters. In work 10 years ago I showed a variety of ways to use continuous parameterizations to yield high-quality approximate solutions to learning problems that involve searching over graph and hyper-graph structures. These continuous relaxations gave enormous speedups over previous approaches based on discrete parameterizations, and we will develop new methods like these to address tuning model hyper-parameters. There is a huge potential impact for the proposed research, with potential applications ranging from medicine to scientific discovery to self-driving cars. In ML we want to build algorithms and models that work across many applications (we want to "build a better hammer" that can be used for many tasks). Thus, breakthroughs on the algorithms underlying ML models immediately impact many applications that use ML (or will use it in the future).