CAREER: Blessing of Nonconvexity in Machine Learning - Landscape Analysis and Efficient Algorithms

职业：机器学习中非凸性的祝福 - 景观分析和高效算法

• 批准号: 2337776
• 负责人: Salar Fattahi
• 金额: $ 63.55万
• 依托单位: Regents of the University of Michigan - Ann Arbor
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-06-01 至 2029-05-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2337776&HistoricalAwards=false
• 关键词: CAREER; Blessing; Nonconvexity; Machine; Learning

The tractability of an optimization problem is often assessed by whether it can be written as a convex program. Yet recent years have witnessed a shift in perspective on what is deemed tractable in optimization: with nonconvex models being used almost exclusively in modern machine learning (ML), it has become increasingly clear that convexity can be traded for representation or flexibility. However, harnessing this power comes at steep costs. First, classical optimization theory asserts that in the absence of convexity, efficient large-scale algorithms generate solutions that may not enjoy any optimality guarantee, which can be detrimental in safety-critical applications of ML. Second, many modern nonconvex optimization problems are overwhelmingly large with outrageously high computational costs. This voracious appetite for computing power makes it difficult to unlock the full representation power of nonconvex models, especially in domains that lack access to substantial computing resources. The goal of this project is to lower the above costs by designing reliable and efficient computational methods for training nonconvex models in ML. In particular, this project aims to uncover the distinct structures of the nonconvex problems in ML that make them tractable, ultimately transmuting nonconvexity from a curse to a blessing. The project will integrate a variety of educational programs for K-12, undergraduate, and graduate students. Notably, new partnerships will be forged with under-resourced schools to help introduce new college opportunities to students from low-income families. To broaden the impact of these programs, the experiences will be shared with different communities in the form of short articles. Furthermore, all the materials will be made available for public use.This project aims to bridge a longstanding gap between optimization and statistical learning. While modern statistical learning favors nonconvex models for their favorable generalization and representation properties, classical optimization theory argues that practical algorithms inevitably struggle to recover globally optimal solutions in nonconvex scenarios. This project challenges the conventional paradigm that evaluates the performance of optimization algorithms solely based on their ability to find global optima. In fact, this project will assert that numerous practical nonconvex models in ML, from low-rank matrix recovery to deep neural networks, possess local solutions that are not only more tractable to obtain than their global counterparts, but also closer to the true solutions, yielding smaller generalization errors. This project aims to formalize this fundamental insight by conducting a systematic analysis of the optimization landscape of nonconvex models around the true solutions, and designing reliable and efficient algorithms to solve them in meaningful settings and scales.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

通常可以通过将其写入凸面程序来评估优化问题的障碍性。然而，最近几年见证了对优化中认为可以说的东西的观点的转变：在现代机器学习（ML）中，非凸模型几乎仅使用，因此已经越来越清楚地表明，可以将凸性交易以供表示或灵活性。但是，利用这种力量的成本高昂。首先，经典优化理论断言，在没有凸度的情况下，有效的大规模算法会生成可能无法享有任何最佳保证的解决方案，这在ML的安全至关重要的应用中可能有害。其次，许多现代的非凸优化问题绝大多数是大量的，并且计算成本非常高。这种对计算能力的狂热胃口使得很难解锁非convex模型的完整表示能力，尤其是在无法获得实质性计算资源的域中。该项目的目的是通过设计可靠，高效的计算方法来降低上述成本，以培训ML中的非凸模型。特别是，该项目旨在揭示ML中非概念问题的独特结构，这些结构使它们具有易于处理，最终将非概念从诅咒转变为祝福。该项目将为K-12，本科生和研究生整合各种教育计划。值得注意的是，新的伙伴关系将与资源不足的学校建立，以帮助向来自低收入家庭的学生介绍新的大学机会。为了扩大这些计划的影响，这些经验将以短篇文章的形式与不同的社区共享。此外，所有材料都将用于公共用途。该项目旨在弥合优化和统计学习之间的长期差距。尽管现代统计学习有利于非凸模型的概括和表示特性，但经典优化理论认为，实际算法不可避免地难以在非凸场中恢复全球最佳解决方案。该项目挑战了传统的范式，该范式仅根据其找到全球最佳功能的能力来评估优化算法的性能。实际上，该项目将断言ML中的许多实用非凸模型，从低级别矩阵恢复到深层神经网络，具有局部解决方案，这些解决方案不仅比其全球对应物更容易获得，而且还更接近真正的解决方案，从而产生了较小的概括错误。该项目旨在通过对非convex模型的优化格局进行系统分析，并设计可靠，有效的算法以在有意义的环境和规模上解决它们。该奖项反映了NSF的法定任务，并通过评估基金会的范围，该奖项值得一提。