Collaborative Research: New Perspectives on Deep Learning: Bridging Approximation, Statistical, and Algorithmic Theories

合作研究：深度学习的新视角：桥接近似、统计和算法理论

• 批准号: 2134077
• 负责人: Guergana Petrova
• 金额: $ 22.5万
• 依托单位: Texas A&M University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-11-01 至 2024-10-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2134077&HistoricalAwards=false
• 关键词: Collaborative; Research; New; Perspectives; Deep

Deep Learning (DL) has led to a renaissance in neural network methods in data-driven science and engineering. The development of DL systems and applications, including computer vision and natural language understanding, has been led primarily by experiments and engineering practice. Mathematical analysis has only begun to provide insights into these complex machine learning systems. The lack of basic understanding has contributed to serious challenges and shortcomings ranging from the fragility and susceptibility to corrupted data to their uninterpretable behaviors. These problems can be traced to fundamental gaps in the mathematical understanding of DL. This project tackles this challenge by bringing approximation, statistical, and algorithmic theories together to develop new mathematical foundations for DL. The goals of the project are to mathematically characterize the strengths and limitations of DL models, and to understand the properties of DL models trained using examples of desired behavior (training data) as well as the tradeoffs between the performance of DL systems and the training dataset size. While DL is already in widespread use, the continued success of DL requires far more complete mathematical understandings and principled approaches to guide its use and reliable application. The project will provide practitioners with clearer guidance on the strengths, limitations, and best approaches to using DL. Broader impacts of the project also include education and mentoring, including the training of graduate students in mathematical fields such as approximation theory, signal processing, statistics, and machine learning and, most importantly, how these fields collectively inform the theory and practice of DL.DL seeks to learn an unknown function from data using compositions (layers) of linear combinations of simple functions (neurons). The shortcomings of DL can be traced to fundamental gaps in its mathematical theory including the following issues. The function spaces that capture the salient properties of DL applications are poorly understood. The characteristics of functions learned through neural network training are mysterious. The ability of DL models to discriminate between data distributions has not yet been quantified satisfactorily. Understanding of the tradeoffs between accuracy and training set size is lacking. This project tackles these challenges by bringing approximation, statistical, and algorithmic theories together to develop new theoretical foundations for DL. This project builds innovative bridges between approximation theory, nonparametric statistics, learning theory and algorithms to develop new mathematical foundations for DL. This includes the development of new model classes of functions that are naturally suited to characterize the properties, strengths, and limitations of deep neural network architectures and applications; novel approaches to understand the roles of regularization and sparsity in DL; fundamental frameworks to quantify the discrimination power of DL and generalized adversarial networks; and innovative theory to make DL algorithms more data efficient through the use of side-information, partial differential equations, and richer forms of data than the conventional function evaluations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度学习导致了神经网络方法在数据驱动的科学和工程领域的复兴。数字图书馆系统和应用的发展，包括计算机视觉和自然语言理解，主要是由实验和工程实践引领的。数学分析才刚刚开始为这些复杂的机器学习系统提供见解。缺乏基本的理解造成了严重的挑战和缺陷，从数据的脆弱性和易感性到数据的无法解释的行为。这些问题可以追溯到对数字逻辑的数学理解上的根本差距。这个项目通过将近似、统计和算法理论结合在一起来为数字图书馆开发新的数学基础来应对这一挑战。该项目的目标是从数学上描述DL模型的优点和局限性，并了解使用期望行为的示例(训练数据)训练的DL模型的性质，以及DL系统的性能和训练数据集大小之间的权衡。虽然数字逻辑已经被广泛使用，但数字逻辑的持续成功需要更全面的数学理解和原则性的方法来指导其使用和可靠的应用。该项目将为实践者提供关于使用数字图书馆的优势、局限性和最佳方法的更清晰的指导。该项目的更广泛影响还包括教育和指导，包括对研究生进行数学领域的培训，如逼近理论、信号处理、统计学和机器学习，最重要的是，这些领域如何共同为DL.DL的理论和实践提供信息。DL寻求使用简单函数(神经元)的线性组合的组合(层)从数据中学习未知函数。数字学习的缺点可以追溯到它的数学理论中的基本空白，包括以下问题。捕捉DL应用程序显著属性的函数空间知之甚少。通过神经网络训练学习的函数的特征是神秘的。DL模型区分数据分布的能力尚未得到令人满意的量化。缺乏对精确度和训练集大小之间的权衡的理解。这个项目通过将近似、统计和算法理论结合在一起来为数字学习开发新的理论基础来解决这些挑战。该项目在近似理论、非参数统计、学习理论和算法之间建立了创新的桥梁，为数字学习奠定了新的数学基础。这包括开发自然适合表征深度神经网络结构和应用的属性、优势和局限性的新的函数模型类；理解正则化和稀疏性在数字图书馆中的作用的新方法；量化数字图书馆和广义对抗性网络的辨别力的基本框架；以及通过使用辅助信息、偏微分方程和比传统函数评估更丰富的数据形式来使数字图书馆算法更有数据效率的创新理论。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Lipschitz widths

利普希茨宽度

• 发表时间: 2022
• 期刊: Constructive approximation
• 影响因子: 2.7
• 作者: G. Petrova, P. Wojtaszczyk

On the entropy numbers and the Kolmogorov widths

• DOI: 10.48550/arxiv.2203.00605
• 发表时间: 2022-02
• 期刊: ArXiv
• 作者: G. Petrova;P. Wojtaszczyk

Neural networks: deep, shallow, or in between?

神经网络：深层、浅层还是介于两者之间？

• 发表时间: 2023
• 期刊: Explorations in the Mathematics of Data Science - The Inaugural Volume of the Center for Approximation and Mathematical Data Analytics
• 作者: G. Petrova, P.Wojtaszczyk

Neural Network Approximation of Refinable Functions

可求函数的神经网络逼近

• DOI: 10.1109/tit.2022.3199601
• 发表时间: 2022
• 期刊: IEEE Transactions on Information Theory
• 影响因子: 2.5
• 作者: Daubechies, Ingrid;DeVore, Ronald;Dym, Nadav;Faigenbaum-Golovin, Shira;Kovalsky, Shahar Z.;Lin, Kung-Chin;Park, Josiah;Petrova, Guergana;Sober, Barak
• 通讯作者: Sober, Barak

Optimal Learning

• DOI: 10.1287/educ.1080.0039
• 发表时间: 2021
• 期刊: Encyclopedia of Evolutionary Psychological Science
• 作者: Warrren B Powell;P. Frazier