Collaborative Research: RI: Medium: Principles for Optimization, Generalization, and Transferability via Deep Neural Collapse

深度学习在工程和科学的各个领域都表现出了前所未有的性能。然而，对他们成功的理论理解仍然难以捉摸。最近，研究人员在学习的特征和分类器中发现并描述了一种优雅的数学结构，称为神经崩溃。这种现象在各种不同的网络架构、数据集和数据域中持续存在。该项目将利用神经崩溃的对称性来开发一个严格的数学理论，以解释它何时以及为什么发生，以及如何使用它来量化泛化性能，并为理解和提高可移植性提供指导。通过推进深度学习的数学基础，该项目预计不仅会影响机器学习社区，还会影响优化、信号和图像处理以及自然语言处理等相关领域。该项目还包括一个综合的外展和教育计划，包括促进K-12学生对计算和STEM概念的可访问性和意识，该项目将扩大我们对训练深度学习模型的非凸优化背后原理的理解，并提供关于其泛化和可移植性的新数学见解，从而产生实际影响。特别是，该项目专注于以下三个总体研究重点：（i）提供一个统一的框架来分析通过一般损失函数训练深度和过度参数化模型到神经崩溃状态的收敛保证，首先是简化的情况，然后是表现出渐进神经崩溃的更一般的深度模型，具有多标签和数据不平衡;（ii）利用神经崩溃的结构，通过表征所得分类器的结构及其对训练数据的轻度依赖，以及通过做出自然分布假设，为深度模型提供更严格的泛化边界;（iii）利用渐进神经崩溃到新环境的泛化来理解深度模型到新领域和任务的可转移性，并制定原则性的方法，以提高可转移性和有效的罚款，该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值进行评估来支持和更广泛的影响审查标准。

Deep learning has demonstrated unprecedented performance across various domains in engineering and science. However, the theoretical understanding of their success has remained elusive. Very recently, researchers discovered and characterized an elegant mathematical structure within the learned features and classifiers called Neural Collapse. This phenomenon persists across a variety of different network architectures, datasets, and data domains. This project will leverage the symmetry of Neural Collapse to develop a rigorous mathematical theory to explain when and why it happens and how it can be used to quantify generalization performance and provide guidelines to understand and improve transferability. By advancing the mathematical foundations of deep learning, this project is expected to influence not only the machine learning community, but also related areas such as optimization, signal and image processing, and natural language processing. The project also involves an integrated outreach and education plan, including promoting accessibility and awareness of computing and STEM concepts for K-12 students.This project will expand our understanding of the principles behind non-convex optimization of training deep learning models, and provide new mathematical insights on their generalization and transferability properties, leading to practical implications. In particular, the project is focused on the following three overarching research thrusts: (i) provide a unified framework to analyze convergence guarantees for training deep and overparametrized models through general loss functions to states of neural collapse, first for simplified cases and then for more general deep models that exhibit progressive neural collapse, with multi-labels and data imbalance; (ii) harness the structure of neural collapse to provide tighter generalization bounds for deep models, by characterizing the structure of the resulting classifiers and their mild dependence on the training data, as well as by making natural distributional assumptions; (iii) leverage the generalization of progressive neural collapse to new environments to understand transferability of deep models to new domains and tasks, and develop principled approaches for improving transferability and efficient fine-tuning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.