Collaborative Research: CCSS: Learning to Optimize: From New Algorithms to New Theory

解决机器学习（ML）问题需要高效和可扩展的优化算法。最先进的通用算法通常需要计算大量的迭代，因此对实时应用的适用性有限。为了克服这个缺点，学习优化（lgo）方法的目标是在元训练的任务分布上学习更短（即更快）的优化路径，基于任务的共同结构和更全局的几何视图，然后将学习到的优化器应用于元测试的新的类似任务。尽管在经验上取得了广泛的成功，但现有的L2O方法主要在具有相似结构的优化任务上表现良好，但在非分布任务上可能表现不佳。此外，对L2O算法的收敛性和泛化性的理论认识很少。因此，所提出的方案将设计新颖的L2O方法，使训练后的优化器可以泛化到广泛的实际任务，特别是分布外任务，并将在L2O训练和测试中保证收敛和泛化性能。具体而言，该计划将设计新的L2O方法，同时具有对分布外任务的泛化性和保证最坏情况收敛的保护特征，将开发用于分析L2O元训练收敛速度的理论框架，并将提供L2O元测试泛化性能的综合表征。新的算法和理论将在物联网（IoT）系统中的设备上模型自适应应用、图像和无线信号的稀疏恢复以及通信系统重构中的算法自适应方面进行评估。该项目预计将大大成熟L2O领域，并在优化、机器学习、信号处理和数据科学的新交叉点为不同群体的学生提供培训机会。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Solving machine learning (ML) problems requires efficient and scalable optimization algorithms. State-of-the-art general purpose algorithms often need to compute a large number of iterations and hence have limited applicability to real-time applications. To circumvent this shortcoming, learning to optimize (L2O) methods aim to learn a shorter (i.e., faster) optimization path over a task distribution at meta-training, based on the tasks’ common structures and a more global view of their geometries, and then apply the learned optimizer to new similar tasks at meta-testing. Despite extensive empirical success, the existing L2O methods perform well mainly on optimization tasks with similar structures, but likely perform poorly on out-of-distribution tasks. Furthermore, there has been little theory understanding the convergence and generalization of L2O algorithms. Thus, the proposed program will design novel L2O approaches, so that the trained optimizer can generalize to a broad range of practical tasks, particularly out-of-distribution tasks, and will have guaranteed convergence and generalization performance in L2O training and testing.Specifically, the proposed program will design new L2O approaches with both generalizability to out-of-distribution tasks and safeguarded feature for guaranteed worst-case convergence, will develop a theoretical framework for analyzing the convergence rate for L2O meta-training, and will provide comprehensive characterization of the generalization performance for L2O meta-testing. The new algorithms and theory will be evaluated over applications of on-device model adaptation in internet-of-things (IoT) systems, sparse recovery for images and wireless signals, and algorithmic adaptation in reconfiguration of communication systems. The project is anticipated to significantly mature the field of L2O, and provide training opportunities for a diverse group of students at the new intersection of optimization, machine learning, signal processing, and data science.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.