RI: Small: Learning to Optimize: Designing and Improving Optimizers by Machine Learning Algorithms

优化的目标是找到使目标函数最小化的最佳参数。现有的优化器通常是由人类设计的，在面对更复杂的问题时往往不够好。例如，当大规模训练深度神经网络时，现有的优化器需要大量的调整，可能找不到一个好的解决方案。人类很难设计出完美的优化器，但机器能否根据解决许多不同问题的经验自动设计出优化器？为了回答这个问题，该项目研究了如何使用机器学习来自动设计优化器，以及如何通过机器学习来改进现有的优化器。这个新的优化器家族将广泛适用于整个数据科学。开发的算法和评估平台将用于刺激这一新的研究领域的未来工作。该项目通过招募多元化的团队，并将研究成果纳入加州大学洛杉矶分校的课程，支持教育和多样性。这个项目的目标是使用机器学习（ML）来改进和自动化现有的优化算法。该项目特别关注两类方法：机器学习优化器和机器辅助优化器。对于机器学习优化器，将更新规则建模为从经验中学习参数的神经网络，并进行一系列研究以确保设计的有效性和可靠性。对于机器学习辅助优化器，开发了机器学习算法来改进现有的优化器，包括批选择、学习率调度和自动超参数调优。开发了一个统一和全面的评估框架，通过对其可扩展性，效率，鲁棒性和各种计算预算下的性能进行基准测试来评估现有和新开发的优化器。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The goal of optimization is to find the best parameters to minimize an objective function. Existing optimizers are typically designed by humans and are often not good enough when facing more complex problems. For example, when training deep neural networks at scale, existing optimizers require a lot of tuning and may not find a good solution. It is hard for humans to design a perfect optimizer, but can a machine automatically design an optimizer based on the experiences on solving many different problems? To answer this question, the project investigates how to use machine learning to automatically design optimizers, and how to improve existing optimizers by machine learning. This new family of optimizers will be broadly applicable across the whole of data science. The developed algorithms and evaluation platforms will be made available to stimulate future work in this new research area. The project supports education and diversity through the recruitment of a diverse team, and incorporation of research results into courses at UCLA. The goal of this project is to use Machine Learning (ML) to improve and automate existing optimization algorithms. In particular, the project focuses on two families of approaches: ML-learned optimizers and ML-assisted optimizers. For ML-learned optimizers, the update rule is modeled as a neural network with parameters learned from experience, and a series of studies are conducted to ensure the effectiveness and soundness of the designs. For ML-assisted optimizers, machine learning algorithms are developed to improve existing optimizers in terms of batch selection, learning rate scheduling, and automatic hyper-parameter tuning. A unified and comprehensive evaluation framework is developed to evaluate existing and newly developed optimizers by benchmarking their scalability, efficiency, robustness, and the performance under various computation budgets.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.