CAREER: Learning Optimization Algorithms from Data: Interpretability, Reliability, and Scalability

高效、可扩展的优化算法(又名优化器)是几乎所有计算领域的基石。在许多优化的实际应用中，人们将在特定的数据分布上重复执行特定类型的优化任务。学习优化(L2O)是一种新兴的范式，它通过学习过去一系列优化任务的表现来自动开发一种优化方法(优化器)。然后，在解决新的但相似的优化任务时，学习的优化器可以展示许多有希望的好处，包括更快的收敛速度和/或更好的解质量。作为一个快速发展的新领域，在L2O的理论基础和实践适用性方面仍存在许多悬而未决的挑战。特别是，习得的优化器通常很难解释、信任和扩展。该项目针对这些研究空白，并扩展到与L2O基础有关的中长期研究方向。具体地说，该项目提出了一个多管齐下的研究议程，包括：一个新的符号表示，使L2O变得更轻和更易理解；一个贝叶斯L2O建模框架，可以量化优化器的不确定性；新的定制设计的L2O模型架构和正则化程序，可以稳健地编码特定问题的先验；以及一个通用的合并方案，将L2O培训与作为教师的经典优化器联系起来。每个推力都涉及L2O的一个独特方面(表示、校准、模型设计和训练策略)。同时，这些推力相互兼容，可以一起应用。拟议的努力结合了深度学习、符号学习、贝叶斯优化和元学习的尖端技术进步。成功的结果有望使L2O成为有原则的科学，以及真正应用的成熟工具。该项目有一个成果传播、教育和推广的综合计划。特别是，该项目产生的所有新算法将被整合到由PI的小组开发和维护的Open-L2O软件包中。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Efficient and scalable optimization algorithms (a.k.a., optimizers) are the cornerstone of almost all computational fields. In many practical applications of optimization, one will repeatedly perform a certain type of optimization tasks over a specific distribution of data. Learning to optimize (L2O) is an emerging paradigm that automatically develops an optimization method (optimizer) by learning from its performance on a set of past optimization tasks. Then on solving new but similar optimization tasks, the learned optimizer can demonstrate many promising benefits including faster convergence and/or better solution quality. As a fast-growing new field, many open challenges remain concerning both L2O's theoretical underpinnings and its practical applicability. In particular, the learned optimizers are often hard to interpret, trust, and scale.The project targets those research gaps and expands to mid-term and long-term research directions pertaining to the foundations of L2O. Specifically, the project proposes a multi-pronged research agenda including: a novel symbolic representation that makes L2O lightweight and more interpretable; a Bayesian L2O modeling framework that can quantify optimizer uncertainty; new customized designs of L2O model architectures and regularizers that can robustly encode problem-specific priors; and a generic amalgamation scheme to bridge L2O training to classical optimizers as teachers. Each thrust addresses a unique aspect of L2O (representation, calibration, model design, and training strategy). Meanwhile, those thrusts are compatible with each other and can be applied together. The proposed efforts synergize cutting-edge technical advances from deep learning, symbolic learning, Bayesian optimization, and meta learning. Successful outcomes are expected to turn L2O into principled science as well as a mature tool for real applications. This project has an integrated plan of result dissemination, education, and outreach. In particular, all new algorithms resulting from the project will be integrated into the Open-L2O software package, developed and maintained by the PI's group.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.