Collaborative Research: RI: Medium: Principles for Optimization, Generalization, and Transferability via Deep Neural Collapse

深度学习在工程和科学的各个领域都表现出了前所未有的表现。然而，对他们成功的理论理解仍然难以捉摸。最近，研究人员在习得的特征和分类器中发现并表征了一种优雅的数学结构，称为神经崩溃。这种现象在各种不同的网络架构、数据集和数据域中持续存在。这个项目将利用神经崩溃的对称性来开发一种严格的数学理论来解释它发生的时间和原因，以及如何使用它来量化泛化性能，并提供了解和改进可转移性的指导方针。通过推进深度学习的数学基础，该项目预计不仅将影响机器学习领域，还将影响优化、信号和图像处理以及自然语言处理等相关领域。该项目还包括一个综合外展和教育计划，包括提高K-12学生对计算和STEM概念的易用性和认识。该项目将扩大我们对训练深度学习模型非凸优化背后的原理的理解，并就其普适性和可转移性提供新的数学见解，从而产生实际意义。具体而言，该项目侧重于以下三个主要研究方向：(I)通过对神经崩溃状态的一般损失函数，首先对于简化情况，然后对于表现出渐进性神经崩溃、多标签和数据不平衡的更一般的深层模型，提供一个统一的框架来分析训练深度模型和总体参数模型的收敛保证；(Ii)利用神经崩溃的结构，通过表征所得到的分类器的结构及其对训练数据的轻微依赖，以及通过做出自然分布假设，来为深层模型提供更紧密的泛化界限；(Iii)利用渐进性神经崩溃到新环境的泛化，以了解深层模型到新领域和任务的可转移性，并开发原则性方法来提高可转移性和高效微调。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Deep learning has demonstrated unprecedented performance across various domains in engineering and science. However, the theoretical understanding of their success has remained elusive. Very recently, researchers discovered and characterized an elegant mathematical structure within the learned features and classifiers called Neural Collapse. This phenomenon persists across a variety of different network architectures, datasets, and data domains. This project will leverage the symmetry of Neural Collapse to develop a rigorous mathematical theory to explain when and why it happens and how it can be used to quantify generalization performance and provide guidelines to understand and improve transferability. By advancing the mathematical foundations of deep learning, this project is expected to influence not only the machine learning community, but also related areas such as optimization, signal and image processing, and natural language processing. The project also involves an integrated outreach and education plan, including promoting accessibility and awareness of computing and STEM concepts for K-12 students.This project will expand our understanding of the principles behind non-convex optimization of training deep learning models, and provide new mathematical insights on their generalization and transferability properties, leading to practical implications. In particular, the project is focused on the following three overarching research thrusts: (i) provide a unified framework to analyze convergence guarantees for training deep and overparametrized models through general loss functions to states of neural collapse, first for simplified cases and then for more general deep models that exhibit progressive neural collapse, with multi-labels and data imbalance; (ii) harness the structure of neural collapse to provide tighter generalization bounds for deep models, by characterizing the structure of the resulting classifiers and their mild dependence on the training data, as well as by making natural distributional assumptions; (iii) leverage the generalization of progressive neural collapse to new environments to understand transferability of deep models to new domains and tasks, and develop principled approaches for improving transferability and efficient fine-tuning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.