Hyper-fast hyper-parameter tuning for the next generation of machine learning

机器学习（ML）使用大量数据来设置模型的参数，并且在越来越多的应用中取得了成功，从语音识别到计算机视觉再到语言翻译。但是ML有一个“超参数”的问题，即影响学习算法和模型结构的变量。我们通常仍然需要花费大量的时间来调整超参数，以获得良好的性能。最近有人强调，调整当前ML语言模型的成本相当于五辆汽车在其整个生命周期中的碳排放量。这种情况只会变得更糟，因为下一代模型将具有更多的超参数。如果我们不解决这个问题，我们将无法解决许多重要问题。该研究计划的长期目标是开发能够在短时间内在大量数据集上训练ML模型的算法。在过去的6年里，我们一直专注于开发收敛速度更快的算法，但我们现在需要转向处理超参数的紧迫问题。该项目的短期目标是专注于解决与超参数的两个典型来源相关的问题：（A）用于进行学习的算法通常具有超参数，例如学习率。(B)正在使用的模型通常具有超参数，例如深度学习模型的深度。我们已经在（A）方面取得了进展。在2019年，我们给出了第一个在训练过程中自动调整最重要的超参数之一--学习率的方法。这种方法保证至少执行以及现代“过参数化”模型的最佳固定学习率。我们计划开发对其他学习超参数不敏感并且不需要过度参数化假设的算法。 我的实验室在解决问题（B）方面具有独特的优势。当前调整网络架构参数的策略往往使用超参数的离散参数化。在10年前的工作中，我展示了各种使用连续参数化来产生高质量近似解的方法，以解决涉及搜索图和超图结构的学习问题。这些连续松弛比以前基于离散参数化的方法提供了巨大的加速，我们将开发这样的新方法来解决调整模型超参数的问题。这项研究具有巨大的潜在影响，其潜在应用范围从医学到科学发现再到自动驾驶汽车。在机器学习中，我们希望构建跨许多应用程序工作的算法和模型（我们希望“构建一个更好的锤子”，可以用于许多任务）。因此，ML模型基础算法的突破会立即影响许多使用ML（或将来使用ML）的应用程序。

Machine learning (ML) uses large amounts data to set the parameters of a model, and is having success in a growing number of applications from speech recognition to computer vision to language translation. But ML has a problem with "hyper-parameters", the variables affecting the learning algorithm and the structure of the model. We typically still need to spend enormous amounts of time tuning hyper-parameters in order to obtain good performance. It was recently highlighted that the cost of tuning current ML language models has a comparable carbon output to five cars throughout their lifetime. This situation will only get worse as the next generation of models will have far more hyper-parameters. We will not be able to solve many important problems until we address this issue. The long-term goal of this research program is to develop algorithms that can train ML models on enormous datasets in a short amount of time. During the last 6 years we have focused on developing algorithms that converge faster, but we now need to turn to the pressing issue of dealing with the hyper-parameters. The short-term goal of this project is to focus on addressing the issues associated with the two typical sources of hyper-parameters: (A) The algorithm used to do the learning typically has hyper-parameters, such as the learning rate. (B) The model that is being used typically has hyper-parameters, such as the depth of a deep learning model. We have already made progress on (A). In 2019 we gave the first method that automatically tunes one of the most important hyper-parameters, the learning rate, during training. This method is guaranteed to perform at least as well as the best fixed learning rate for modern "over-parameterized" models. We plan to develop algorithms that are insensitive to other learning hyper-parameters and that do not require the over-parameterized assumption. My lab is uniquely positioned to address problem (B). Current strategies for tuning the parameters of network architectures tend to use discrete parameterizations of the hyper-parameters. In work 10 years ago I showed a variety of ways to use continuous parameterizations to yield high-quality approximate solutions to learning problems that involve searching over graph and hyper-graph structures. These continuous relaxations gave enormous speedups over previous approaches based on discrete parameterizations, and we will develop new methods like these to address tuning model hyper-parameters. There is a huge potential impact for the proposed research, with potential applications ranging from medicine to scientific discovery to self-driving cars. In ML we want to build algorithms and models that work across many applications (we want to "build a better hammer" that can be used for many tasks). Thus, breakthroughs on the algorithms underlying ML models immediately impact many applications that use ML (or will use it in the future).