Collaborative Research: RI: Medium: Principles for Optimization, Generalization, and Transferability via Deep Neural Collapse

深度学习在工程和科学的各个领域都表现出前所未有的性能。然而，对他们成功的理论理解仍然难以捉摸。最近，研究人员在学习特征和分类器中发现并描述了一个优雅的数学结构，称为神经崩溃。这种现象在各种不同的网络体系结构、数据集和数据域中持续存在。该项目将利用神经崩溃的对称性来发展一个严格的数学理论来解释它何时和为什么发生，以及如何使用它来量化泛化性能，并为理解和提高可转移性提供指导。通过推进深度学习的数学基础，该项目预计不仅会影响机器学习社区，还会影响优化、信号和图像处理以及自然语言处理等相关领域。该项目还涉及一项综合外展和教育计划，包括促进K-12学生对计算机和STEM概念的可及性和认识。该项目将扩展我们对训练深度学习模型的非凸优化背后原理的理解，并为其泛化和可转移性提供新的数学见解，从而产生实际意义。特别是，该项目侧重于以下三个总体研究重点：(i)提供一个统一的框架，通过神经崩溃状态的一般损失函数来分析训练深度和过度参数化模型的收敛保证，首先用于简化情况，然后用于表现出渐进神经崩溃的更一般的深度模型，具有多标签和数据不平衡；（ii）通过描述结果分类器的结构及其对训练数据的轻微依赖，以及通过做出自然分布假设，利用神经崩溃的结构为深度模型提供更严格的泛化界限；（iii）利用渐进式神经崩溃在新环境中的泛化来理解深度模型在新领域和任务中的可转移性，并开发原则性方法来提高可转移性和有效的微调。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Deep learning has demonstrated unprecedented performance across various domains in engineering and science. However, the theoretical understanding of their success has remained elusive. Very recently, researchers discovered and characterized an elegant mathematical structure within the learned features and classifiers called Neural Collapse. This phenomenon persists across a variety of different network architectures, datasets, and data domains. This project will leverage the symmetry of Neural Collapse to develop a rigorous mathematical theory to explain when and why it happens and how it can be used to quantify generalization performance and provide guidelines to understand and improve transferability. By advancing the mathematical foundations of deep learning, this project is expected to influence not only the machine learning community, but also related areas such as optimization, signal and image processing, and natural language processing. The project also involves an integrated outreach and education plan, including promoting accessibility and awareness of computing and STEM concepts for K-12 students.This project will expand our understanding of the principles behind non-convex optimization of training deep learning models, and provide new mathematical insights on their generalization and transferability properties, leading to practical implications. In particular, the project is focused on the following three overarching research thrusts: (i) provide a unified framework to analyze convergence guarantees for training deep and overparametrized models through general loss functions to states of neural collapse, first for simplified cases and then for more general deep models that exhibit progressive neural collapse, with multi-labels and data imbalance; (ii) harness the structure of neural collapse to provide tighter generalization bounds for deep models, by characterizing the structure of the resulting classifiers and their mild dependence on the training data, as well as by making natural distributional assumptions; (iii) leverage the generalization of progressive neural collapse to new environments to understand transferability of deep models to new domains and tasks, and develop principled approaches for improving transferability and efficient fine-tuning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.