CAREER: From Shallow to Deep Representation Learning: Global Nonconvex Optimization Theories and Efficient Algorithms

职业：从浅层到深层表示学习：全局非凸优化理论和高效算法

• 批准号: 2143904
• 负责人: Qing Qu
• 金额: $ 63.33万
• 依托单位: Regents of the University of Michigan - Ann Arbor
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-05-01 至 2027-04-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2143904&HistoricalAwards=false
• 关键词: CAREER; Shallow; Deep; Representation; Learning

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2). Machine learning is by now transforming many fields of science and engineering. However, as data sets continuously increase in both volume and dimensions, the performance of modern machine-learning methods has become critically dependent on the data representations being used. In the past decade, although many deep-representation learning methods have enjoyed remarkable empirical success, the underlying principles behind this success have largely remained a mystery, a situation which has hindered further development and broader adoption. A major difficulty stems from the nonlinearities of the data representation models that often lead to complicated and challenging non-convex optimization problems. This project aims to advance the theoretical foundations from shallow-representation learning (e.g., learning sparsifying dictionaries) to deep-representation learning (e.g., learning deep neural networks) by exploiting the geometric properties of non-convex optimization landscapes and the intrinsic structures of the data. The impact of this research will take the form of new guiding principles for better model/architecture design, optimization and robustness in both supervised and unsupervised scenarios. The research program will be integrated with education activities that include training a diverse workforce in STEM at both undergraduate and graduate levels, designing machine learning modules appropriate for dissemination to K-12 students, and promoting women and underrepresented students through various outreach activities. This project seeks to bridge the gap between the theory and practice of representation learning by developing a principled and unified mathematical framework based on recent developments in global non-convex optimization theory. The mathematical foundations for both shallow and deep representation learning will be advanced by leveraging both the geometric properties of the corresponding non-convex optimization landscapes, and the low-dimensional structures of high-dimensional data. The resulting geometric insights will clarify the kind of representations can be learned through optimization, and will guide the development of efficient and globally convergent training algorithms. The proposed framework will be applied to the study of a wide spectrum of representation learning problems in supervised, self-supervised and unsupervised learning; correspondingly, their generalization and robustness will be investigated by understanding the learned representations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该奖项全部或部分根据2021年美国救援计划法案（公法117-2）资助。机器学习正在改变许多科学和工程领域。然而，随着数据集在数量和维度上的不断增加，现代机器学习方法的性能已经变得严重依赖于所使用的数据表示。在过去的十年中，尽管许多深度表征学习方法在经验上取得了显著的成功，但这种成功背后的基本原理在很大程度上仍然是一个谜，这种情况阻碍了进一步的发展和更广泛的采用。一个主要的困难源于数据表示模型的非线性，这往往导致复杂和具有挑战性的非凸优化问题。该项目旨在推进浅层表征学习的理论基础（例如，学习稀疏化字典）到深度表示学习（例如，学习深度神经网络），利用非凸优化景观的几何属性和数据的内在结构。这项研究的影响将采取新的指导原则的形式，以更好的模型/架构设计，优化和监督和无监督场景的鲁棒性。该研究计划将与教育活动相结合，包括在本科和研究生阶段培训多元化的STEM劳动力，设计适合传播给K-12学生的机器学习模块，并通过各种外展活动促进妇女和代表性不足的学生。该项目旨在通过基于全局非凸优化理论的最新发展开发一个原则性和统一的数学框架，弥合表征学习理论和实践之间的差距。浅层和深层表示学习的数学基础将通过利用相应的非凸优化景观的几何属性和高维数据的低维结构来推进。由此产生的几何见解将澄清可以通过优化学习的表示类型，并将指导高效和全局收敛的训练算法的开发。建议的框架将被应用到监督，自我监督和无监督学习的表征学习问题的广泛研究;相应地，他们的泛化和鲁棒性将通过了解学到的representations.This奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

项目成果

Miniaturizing a Chip-Scale Spectrometer Using Local Strain Engineering and Total-Variation Regularized Reconstruction

• DOI: 10.1021/acs.nanolett.2c02654
• 发表时间: 2022-10-12
• 期刊: NANO LETTERS
• 影响因子: 10.8
• 作者: Sarwar，Tuba;Yaras，Can;Ku，Pei-Cheng
• 通讯作者: Ku，Pei-Cheng

Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold

• DOI: 10.48550/arxiv.2209.09211
• 发表时间: 2022-09
• 期刊: ArXiv
• 作者: Can Yaras;Peng Wang;Zhihui Zhu;L. Balzano;Qing Qu

Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learning

• DOI: 10.48550/arxiv.2210.10041
• 发表时间: 2022-10
• 期刊: ArXiv
• 作者: Shuo Xie;Jiahao Qiu;Ankita Pasad;Li Du;Qing Qu;Hongyuan Mei

Robust Training under Label Noise by Over-parameterization

• 发表时间: 2022-02
• 期刊: ArXiv
• 作者: Sheng Liu;Zhihui Zhu;Qing Qu;Chong You

On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features

• DOI: 10.48550/arxiv.2203.01238
• 发表时间: 2022-03
• 期刊: ArXiv
• 作者: Jinxin Zhou-;Xiao Li;Tian Ding;Chong You;Qing Qu;Zhihui Zhu