CAREER: From Shallow to Deep Representation Learning: Global Nonconvex Optimization Theories and Efficient Algorithms

职业：从浅层到深层表示学习：全局非凸优化理论和高效算法

• 批准号: 2143904
• 负责人: Qing Qu
• 金额: $ 63.33万
• 依托单位: Regents of the University of Michigan - Ann Arbor
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-05-01 至 2027-04-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2143904&HistoricalAwards=false
• 关键词: CAREER; Shallow; Deep; Representation; Learning

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2). Machine learning is by now transforming many fields of science and engineering. However, as data sets continuously increase in both volume and dimensions, the performance of modern machine-learning methods has become critically dependent on the data representations being used. In the past decade, although many deep-representation learning methods have enjoyed remarkable empirical success, the underlying principles behind this success have largely remained a mystery, a situation which has hindered further development and broader adoption. A major difficulty stems from the nonlinearities of the data representation models that often lead to complicated and challenging non-convex optimization problems. This project aims to advance the theoretical foundations from shallow-representation learning (e.g., learning sparsifying dictionaries) to deep-representation learning (e.g., learning deep neural networks) by exploiting the geometric properties of non-convex optimization landscapes and the intrinsic structures of the data. The impact of this research will take the form of new guiding principles for better model/architecture design, optimization and robustness in both supervised and unsupervised scenarios. The research program will be integrated with education activities that include training a diverse workforce in STEM at both undergraduate and graduate levels, designing machine learning modules appropriate for dissemination to K-12 students, and promoting women and underrepresented students through various outreach activities. This project seeks to bridge the gap between the theory and practice of representation learning by developing a principled and unified mathematical framework based on recent developments in global non-convex optimization theory. The mathematical foundations for both shallow and deep representation learning will be advanced by leveraging both the geometric properties of the corresponding non-convex optimization landscapes, and the low-dimensional structures of high-dimensional data. The resulting geometric insights will clarify the kind of representations can be learned through optimization, and will guide the development of efficient and globally convergent training algorithms. The proposed framework will be applied to the study of a wide spectrum of representation learning problems in supervised, self-supervised and unsupervised learning; correspondingly, their generalization and robustness will be investigated by understanding the learned representations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该奖项是根据2021年《美国救援计划法》（公法117-2）全部或部分资助的。机器学习现在可以改变科学和工程的许多领域。但是，随着数据集在数量和尺寸的不断增加时，现代机器学习方法的性能已关键取决于所使用的数据表示。在过去的十年中，尽管许多深入代表性学习方法取得了杰出的经验成功，但这种成功的基本原则在很大程度上仍然是一个谜，这种情况阻碍了进一步的发展和更广泛的采用。一个主要的困难源于数据表示模型的非线性，这些模型通常会导致复杂且具有挑战性的非凸优化问题。该项目旨在通过利用非convex优化景观和数据的内在结构的几何特性，从浅层代表学习（例如，学习稀疏字典）从浅层代表学习（例如学习稀疏字典）转化为深层代表学习（例如学习深神经网络），从而将非代表性的基础从浅层代表学习（例如学习稀疏字典）推进了理论基础。这项研究的影响将采用新的指导原则的形式，以在受监督和无监督的情况下更好地模型/建筑设计，优化和鲁棒性。该研究计划将与教育活动集成，其中包括在本科和研究生级别培训STEM的多样化劳动力，设计适合于K-12学生传播的机器学习模块，以及通过各种外展活动促进女性和代表性不足的学生。该项目试图通过基于全球非凸优化理论的最新发展来开发一个原则和统一的数学框架来弥合代表学习理论和实践之间的差距。通过利用相应的非凸优化景观的几何特性和高维数据的低维结构，将提出浅层和深度表示学习的数学基础。由此产生的几何见解将阐明可以通过优化学习的表示形式，并将指导有效且全球收敛的训练算法的发展。所提出的框架将应用于对受监督，自学和无监督学习的各种代表性学习问题的研究；相应地，将通过理解学习的表示形式来调查它们的概括和鲁棒性。该奖项反映了NSF的法定任务，并使用基金会的知识分子优点和更广泛的影响审查标准，被认为值得通过评估来获得支持。

项目成果

Miniaturizing a Chip-Scale Spectrometer Using Local Strain Engineering and Total-Variation Regularized Reconstruction

• DOI: 10.1021/acs.nanolett.2c02654
• 发表时间: 2022-10-12
• 期刊: NANO LETTERS
• 影响因子: 10.8
• 作者: Sarwar，Tuba;Yaras，Can;Ku，Pei-Cheng
• 通讯作者: Ku，Pei-Cheng

Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold

• DOI: 10.48550/arxiv.2209.09211
• 发表时间: 2022-09
• 期刊: ArXiv
• 作者: Can Yaras;Peng Wang;Zhihui Zhu;L. Balzano;Qing Qu

Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learning

• DOI: 10.48550/arxiv.2210.10041
• 发表时间: 2022-10
• 期刊: ArXiv
• 作者: Shuo Xie;Jiahao Qiu;Ankita Pasad;Li Du;Qing Qu;Hongyuan Mei

On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features

• DOI: 10.48550/arxiv.2203.01238
• 发表时间: 2022-03
• 期刊: ArXiv
• 作者: Jinxin Zhou-;Xiao Li;Tian Ding;Chong You;Qing Qu;Zhihui Zhu

Are All Losses Created Equal: A Neural Collapse Perspective

• DOI: 10.48550/arxiv.2210.02192
• 发表时间: 2022-10
• 期刊: ArXiv
• 作者: Jinxin Zhou-;Chong You;Xiao Li;Kangning Liu;Sheng Liu;Qing Qu;Zhihui Zhu