RI: Small: Learning to Optimize: Designing and Improving Optimizers by Machine Learning Algorithms

优化的目标是找到最佳参数，以最小化目标函数。现有的优化器通常是由人类设计的，在面对更复杂的问题时往往不够好。例如，在大规模训练深度神经网络时，现有的优化器需要大量的调整，可能找不到好的解决方案。人类很难设计出一个完美的优化器，但机器能根据解决许多不同问题的经验自动设计出一个优化器吗？为了回答这个问题，该项目研究了如何使用机器学习来自动设计优化器，以及如何通过机器学习来改进现有的优化器。这个新的优化器家族将广泛适用于整个数据科学。开发的算法和评估平台将用于刺激这一新研究领域的未来工作。该项目通过招募多样化的团队来支持教育和多样性，并将研究成果纳入加州大学洛杉矶分校的课程。该项目的目标是使用机器学习（ML）来改进和自动化现有的优化算法。特别是，该项目侧重于两个家庭的方法：ML学习优化和ML辅助优化。对于ML学习的优化器，更新规则被建模为一个神经网络，参数从经验中学习，并进行了一系列的研究，以确保设计的有效性和合理性。对于ML辅助优化器，机器学习算法被开发以改进现有优化器在批次选择、学习率调度和自动超参数调整方面。一个统一的和全面的评估框架被开发来评估现有的和新开发的优化器，通过基准测试它们的可扩展性，效率，鲁棒性，以及在各种计算预算下的性能。这个奖项反映了NSF的法定使命，并被认为是值得支持的，通过使用基金会的智力价值和更广泛的影响审查标准进行评估。

The goal of optimization is to find the best parameters to minimize an objective function. Existing optimizers are typically designed by humans and are often not good enough when facing more complex problems. For example, when training deep neural networks at scale, existing optimizers require a lot of tuning and may not find a good solution. It is hard for humans to design a perfect optimizer, but can a machine automatically design an optimizer based on the experiences on solving many different problems? To answer this question, the project investigates how to use machine learning to automatically design optimizers, and how to improve existing optimizers by machine learning. This new family of optimizers will be broadly applicable across the whole of data science. The developed algorithms and evaluation platforms will be made available to stimulate future work in this new research area. The project supports education and diversity through the recruitment of a diverse team, and incorporation of research results into courses at UCLA. The goal of this project is to use Machine Learning (ML) to improve and automate existing optimization algorithms. In particular, the project focuses on two families of approaches: ML-learned optimizers and ML-assisted optimizers. For ML-learned optimizers, the update rule is modeled as a neural network with parameters learned from experience, and a series of studies are conducted to ensure the effectiveness and soundness of the designs. For ML-assisted optimizers, machine learning algorithms are developed to improve existing optimizers in terms of batch selection, learning rate scheduling, and automatic hyper-parameter tuning. A unified and comprehensive evaluation framework is developed to evaluate existing and newly developed optimizers by benchmarking their scalability, efficiency, robustness, and the performance under various computation budgets.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.