CAREER: Structured Minimax Optimization: Theory, Algorithms, and Applications in Robust Learning

本提案旨在扩展极大极小优化的数学理论，这是一种非常常用的方法，在推进信息论、机器学习和信号处理领域发挥了关键作用。值得注意的是，最近人们对极大极小优化的兴趣激增，因为它与人工智能（AI）有着重要的相关性，它可以用来使深度神经网络更有弹性地抵御数据底层分布中的对抗性干扰。虽然最近在增强极大极小优化理论及其算法方面取得了进展，但该理论在实际AI场景中的适用性仍然存在显著差距。现有的算法和理论主要集中在所谓的凹凸优化设置上，而当代的学习应用经常涉及不符合这种结构的极大极小问题，并且更加复杂。本项目旨在利用人工智能预测函数的独特结构，为人工智能开发有效的极大极小优化方法。这个跨学科项目将研究成果整合到研究生和本科课程中，促进高中生对STEM的兴趣。该项目的总体目标是通过研究结构化极大极小优化问题，推进鲁棒机器学习模型训练的优化理论和算法。鲁棒学习中的最小化问题是基于预测函数模型构建的，我们将研究不同类型的模型，如神经网络。对于每个考虑的预测函数，极大极小问题的极小化分量是非凸的；尽管如此，它们每个都有一个独特的结构，我们在不同的推力中仔细探索。此外，在每个推力中，我们研究了不同类型的结构化内部最大化问题，这些问题是由本项目中密切研究的三种应用驱动的：分布鲁棒学习，对抗鲁棒训练和离散鲁棒学习。所开发的算法可以在计算成本和精度方面显著推进这些应用领域。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This proposal seeks to extend the mathematical theory of minimax optimization, a very commonly used approach that has played a pivotal role in advancing the fields of information theory, machine learning and signal processing. Notably, there has been a recent surge of interest in minimax optimization due to its critical relevance in artificial intelligence (AI), where it can be used to make deep neural networks more resilient against adversarial disturbances in the underlying distribution of data. While there has been recent progress in enhancing the theory of minimax optimization and its algorithms, a notable gap persists in the applicability of this theory to real-world AI scenarios. Existing algorithms and theory primarily focus on the so-called convex-concave optimization setting, while contemporary learning applications frequently entail minimax problems that do not adhere to this structure and are more complex. This project aims to develop efficient methods for minimax optimization for AI by capitalizing on the unique structure of the AI prediction function. This interdisciplinary project integrates research findings into graduate and undergraduate courses and promotes STEM interest among high school students. The overarching goal of this project is to advance optimization theory and algorithms for robust machine learning model training by investigating their structured minimax optimization problems. The minimization problems in robust learning are structured based on the prediction function model, and we will investigate different types of models, such as neural networks. For each considered prediction function, the minimization component of the minimax problem is nonconvex; nonetheless, they each possess a distinct structure that we meticulously explore in separate thrusts. Moreover, within each thrust, we study different types of structured inner maximization problems motivated by the three applications closely examined in this project: distributionally robust learning, adversarially robust training, and discrete robust learning. The developed algorithms can significantly advance these application areas in terms of computational cost and accuracy.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.