AF: Small: Bridging the Past and Present of Continuous Optimization for Learning

AF：小：连接持续优化学习的过去和现在

• 批准号: 2224213
• 负责人: Stephen Wright
• 金额: $ 60万
• 依托单位: University of Wisconsin-Madison
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-10-01 至 2025-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2224213&HistoricalAwards=false
• 关键词: AF; Small; Bridging; Past; Present

Machine learning (ML) is being used to drive applications of artificial intelligence in many areas of science and society. Ultimately, ML problems must be distilled using statistical and mathematical techniques into problems that can be solved using computational algorithms. For the past 25 years, ML has depended heavily on the field of optimization to provide a wealth of techniques to formulate and solve ML problems. Indeed, the ML problems that optimization is called on to solve continue to grow in complexity and difficulty. Optimization tools have been applied to some of these modern ML problems, but often in ways that lack theoretical guarantees on their performance. This project aims to develop new, powerful, principled optimization approaches for solving these more complex modern ML problems. The project will study several areas of optimization that are critical to current research in ML, developing new algorithmic techniques and new theory for these areas. Priorities include the discovery of algorithms that leverage the structures that characterize various ML problems, such as sparsity, and the development of theoretical analysis to illuminate the computational performance of practical algorithms, including finite-time sample complexity bounds in the presence of nonconvexity. Three specific thrusts of the project include (i) the development of algorithms and analysis techniques for convex-concave min-max problems that take into account sparsity or regularity properties; (ii) the development of theoretically grounded algorithms for solving nonlinear programs with nonconvex functions and stochastic oracles, motivated by problems arising in constrained neural networks, problems with fairness constraints, and distributionally robust optimization; and (iii) advancing the use of optimization in reinforcement learning, such as making use of the primal-dual techniques developed in the first thrust and extending the theory of policy gradient methods to account for the inexactness that inevitably arises in practical implementations. The project's research agenda has foundations in classical optimization and more recent developments in optimization, control, learning theory, and statistics.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

机器学习（ML）被用于推动人工智能在科学和社会的许多领域的应用。最终，机器学习问题必须使用统计和数学技术提炼成可以使用计算算法解决的问题。在过去的25年里，ML在很大程度上依赖于优化领域来提供丰富的技术来制定和解决ML问题。事实上，需要优化来解决的机器学习问题的复杂性和难度不断增加。优化工具已经被应用于这些现代机器学习问题中的一些，但通常缺乏对其性能的理论保证。该项目旨在开发新的、强大的、有原则的优化方法，以解决这些更复杂的现代机器学习问题。该项目将研究对ML当前研究至关重要的几个优化领域，为这些领域开发新的算法技术和新理论。优先事项包括发现利用表征各种ML问题（如稀疏性）的结构的算法，以及理论分析的发展，以阐明实际算法的计算性能，包括存在非凸性时的有限时间样本复杂性界限。该项目的三个具体目标包括：（i）为考虑稀疏性或规律性的凸-凹最小-最大问题开发算法和分析技术;（ii）为解决具有非凸函数和随机预言的非线性规划开发理论基础算法，其动机是约束神经网络中出现的问题、公平约束问题和分布鲁棒优化;以及（iii）推进强化学习中优化的使用，例如利用在第一次推力中开发的原始对偶技术，并扩展策略梯度方法的理论，以解决实际实现中不可避免的不精确性。该项目的研究议程在经典优化和最优化，控制，学习理论和统计学的最新发展的基础。这个奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

项目成果

Beyond the Golden Ratio for Variational Inequality Algorithms

• DOI: 10.48550/arxiv.2212.13955
• 发表时间: 2022-12
• 期刊: ArXiv
• 作者: Ahmet Alacaoglu;A. Böhm;Yura Malitsky

Complexity of a projected Newton-CG method for optimization with bounds

• DOI: 10.1007/s10107-023-02000-z
• 发表时间: 2021-03
• 期刊: Mathematical Programming
• 影响因子: 2.7
• 作者: Yue Xie;Stephen J. Wright

Convergence of First-Order Methods for Constrained Nonconvex Optimization with Dependent Data

具有相关数据的约束非凸优化的一阶方法的收敛性

• 发表时间: 2023
• 期刊: Proceedings of Machine Learning Research
• 作者: Ahmet Alacaoglu;Hanbaek Lyu
• 通讯作者: Hanbaek Lyu