Hyper-fast hyper-parameter tuning for the next generation of machine learning

点击翻译按钮获取中文摘要

Machine learning (ML) uses large amounts data to set the parameters of a model, and is having success in a growing number of applications from speech recognition to computer vision to language translation. But ML has a problem with "hyper-parameters", the variables affecting the learning algorithm and the structure of the model. We typically still need to spend enormous amounts of time tuning hyper-parameters in order to obtain good performance. It was recently highlighted that the cost of tuning current ML language models has a comparable carbon output to five cars throughout their lifetime. This situation will only get worse as the next generation of models will have far more hyper-parameters. We will not be able to solve many important problems until we address this issue. The long-term goal of this research program is to develop algorithms that can train ML models on enormous datasets in a short amount of time. During the last 6 years we have focused on developing algorithms that converge faster, but we now need to turn to the pressing issue of dealing with the hyper-parameters. The short-term goal of this project is to focus on addressing the issues associated with the two typical sources of hyper-parameters: (A) The algorithm used to do the learning typically has hyper-parameters, such as the learning rate. (B) The model that is being used typically has hyper-parameters, such as the depth of a deep learning model. We have already made progress on (A). In 2019 we gave the first method that automatically tunes one of the most important hyper-parameters, the learning rate, during training. This method is guaranteed to perform at least as well as the best fixed learning rate for modern "over-parameterized" models. We plan to develop algorithms that are insensitive to other learning hyper-parameters and that do not require the over-parameterized assumption. My lab is uniquely positioned to address problem (B). Current strategies for tuning the parameters of network architectures tend to use discrete parameterizations of the hyper-parameters. In work 10 years ago I showed a variety of ways to use continuous parameterizations to yield high-quality approximate solutions to learning problems that involve searching over graph and hyper-graph structures. These continuous relaxations gave enormous speedups over previous approaches based on discrete parameterizations, and we will develop new methods like these to address tuning model hyper-parameters. There is a huge potential impact for the proposed research, with potential applications ranging from medicine to scientific discovery to self-driving cars. In ML we want to build algorithms and models that work across many applications (we want to "build a better hammer" that can be used for many tasks). Thus, breakthroughs on the algorithms underlying ML models immediately impact many applications that use ML (or will use it in the future).