CAREER: From Shallow to Deep Representation Learning: Global Nonconvex Optimization Theories and Efficient Algorithms

该奖项的全部或部分资金来自《2021年美国救援计划法案》(公法117-2)。到目前为止，机器学习正在改变许多科学和工程领域。然而，随着数据集在体积和维度上的不断增加，现代机器学习方法的性能已经变得严重依赖于所使用的数据表示法。在过去的十年里，尽管许多深度表征学习方法在经验上取得了显著的成功，但这种成功背后的基本原理在很大程度上仍然是一个谜，这种情况阻碍了进一步的发展和更广泛的采用。一个主要的困难来自于数据表示模型的非线性，这往往导致复杂和具有挑战性的非凸优化问题。该项目旨在通过利用非凸优化景观的几何性质和数据的内在结构，将理论基础从浅表示学习(例如，学习稀疏字典)推进到深度表示学习(例如，学习深层神经网络)。这项研究的影响将以新的指导原则的形式出现，以便在有监督和无监督的情况下更好地进行模型/架构设计、优化和健壮性。该研究计划将与教育活动相结合，包括在本科生和研究生层面培训STEM的多样化劳动力，设计适合向K-12学生传播的机器学习模块，并通过各种外联活动促进女性和代表性不足的学生。这个项目试图通过根据全球非凸优化理论的最新发展制定一个原则性和统一的数学框架来弥合表示学习的理论和实践之间的差距。浅层和深层表示学习的数学基础将通过利用相应非凸优化景观的几何特性和高维数据的低维结构来提高。由此得到的几何洞察力将澄清可以通过优化学习的表示类型，并将指导高效和全局收敛的训练算法的开发。建议的框架将应用于监督、自我监督和非监督学习中广泛的表征学习问题的研究；相应地，将通过理解学习的表征来调查它们的概括性和稳健性。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2). Machine learning is by now transforming many fields of science and engineering. However, as data sets continuously increase in both volume and dimensions, the performance of modern machine-learning methods has become critically dependent on the data representations being used. In the past decade, although many deep-representation learning methods have enjoyed remarkable empirical success, the underlying principles behind this success have largely remained a mystery, a situation which has hindered further development and broader adoption. A major difficulty stems from the nonlinearities of the data representation models that often lead to complicated and challenging non-convex optimization problems. This project aims to advance the theoretical foundations from shallow-representation learning (e.g., learning sparsifying dictionaries) to deep-representation learning (e.g., learning deep neural networks) by exploiting the geometric properties of non-convex optimization landscapes and the intrinsic structures of the data. The impact of this research will take the form of new guiding principles for better model/architecture design, optimization and robustness in both supervised and unsupervised scenarios. The research program will be integrated with education activities that include training a diverse workforce in STEM at both undergraduate and graduate levels, designing machine learning modules appropriate for dissemination to K-12 students, and promoting women and underrepresented students through various outreach activities. This project seeks to bridge the gap between the theory and practice of representation learning by developing a principled and unified mathematical framework based on recent developments in global non-convex optimization theory. The mathematical foundations for both shallow and deep representation learning will be advanced by leveraging both the geometric properties of the corresponding non-convex optimization landscapes, and the low-dimensional structures of high-dimensional data. The resulting geometric insights will clarify the kind of representations can be learned through optimization, and will guide the development of efficient and globally convergent training algorithms. The proposed framework will be applied to the study of a wide spectrum of representation learning problems in supervised, self-supervised and unsupervised learning; correspondingly, their generalization and robustness will be investigated by understanding the learned representations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.