RI: Small: Learning to Optimize: Designing and Improving Optimizers by Machine Learning Algorithms

点击翻译按钮获取中文摘要

The goal of optimization is to find the best parameters to minimize an objective function. Existing optimizers are typically designed by humans and are often not good enough when facing more complex problems. For example, when training deep neural networks at scale, existing optimizers require a lot of tuning and may not find a good solution. It is hard for humans to design a perfect optimizer, but can a machine automatically design an optimizer based on the experiences on solving many different problems? To answer this question, the project investigates how to use machine learning to automatically design optimizers, and how to improve existing optimizers by machine learning. This new family of optimizers will be broadly applicable across the whole of data science. The developed algorithms and evaluation platforms will be made available to stimulate future work in this new research area. The project supports education and diversity through the recruitment of a diverse team, and incorporation of research results into courses at UCLA. The goal of this project is to use Machine Learning (ML) to improve and automate existing optimization algorithms. In particular, the project focuses on two families of approaches: ML-learned optimizers and ML-assisted optimizers. For ML-learned optimizers, the update rule is modeled as a neural network with parameters learned from experience, and a series of studies are conducted to ensure the effectiveness and soundness of the designs. For ML-assisted optimizers, machine learning algorithms are developed to improve existing optimizers in terms of batch selection, learning rate scheduling, and automatic hyper-parameter tuning. A unified and comprehensive evaluation framework is developed to evaluate existing and newly developed optimizers by benchmarking their scalability, efficiency, robustness, and the performance under various computation budgets.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.