Random Matrix Limit Theorems for Deep Neural Networks

深度神经网络的随机矩阵极限定理

• 批准号: RGPIN-2021-02533
• 负责人: Nica, Mihai
• 金额: $ 1.89万
• 依托单位: University of Guelph
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2021
• 资助国家: 加拿大
• 起止时间: 2021-01-01 至 2022-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=742034
• 关键词: Random; Matrix; Limit; Theorems; Deep

Recent advances in deep neural networks (DNNs) have had a tremendous impact on the modern world. However, the theoretical understanding of these systems is still in its infancy. As a matter of course, the research in this area has been empirically driven and computationally focused rather than emphasizing mathematically rigorous results. There are many open theoretical questions which have been uncovered by empirical work that are now ripe for mathematical analysis. I propose a research program that will develop and apply tools from theoretical probability, specifically random matrix theory, to gain a better understanding of the theory of DNNs and other machine learning systems. I will focus on developing new limit theorems which describe behavior when the number of parameters and/or data becomes very large. These results will help us understand how DNNs work and help us design more effective systems in the future. My objectives of the research program include: 1. The neural tangent kernel: A random matrix that explains the behavior of large networks The neural tangent kernel (NTK) is a recently discovered non-random asymptotic object that explains the behavior of DNNs of fixed depth in the infinite width limit, when the number of neurons in each hidden layer tends to infinity. When applied to random data, the NTK gives a random matrix whose dimensions are the number of given data points. Analysis of this random matrix can explain how DNNs behave during training and can be used to understand the generalization error in deep neural networks. I propose to study this model using random matrix theory. 2. Applied free probability: Advanced tools for random matrix analysis The theory of free probability was originally developed in connection to pure problems in the field of operator algebras. More recently however, methods from free probability and its extensions have emerged as powerful tools for computing asymptotic features of complicated random matrix models. One application is to use free probability to compute the limiting spectrum of large random matrix models connected to DNNs. I also plan to investigate the use of operator valued free probability, a powerful extension of free probability, to study block random matrices related to DNNs. 3. Kardar-Parisi-Zhang (KPZ) universality: Fluctuations of random matrix eigenvalues The KPZ universality class is a collection of stochastic systems, including examples from stochastic PDEs and interacting particle systems, which all share the same type of universal asymptotic random behavior. An important application is the behaviour of the largest eigenvalues in many random matrix models. (As opposed to the bulk behavior of the spectrum captured by other random matrix tools). I plan to apply ideas from KPZ to random matrix problems coming from DNNs and other statistical learning models to analyze the evolution of the largest eigenvalues in these problems.

深度神经网络(DNN)的最新进展对现代世界产生了巨大的影响。然而，对这些制度的理论认识还处于起步阶段。理所当然的是，这一领域的研究一直是经验驱动的，注重计算，而不是强调严格的数学结果。有许多开放的理论问题已经被经验工作发现，现在数学分析的时机已经成熟。我提出了一个研究计划，将开发和应用理论概率，特别是随机矩阵理论的工具，以更好地理解DNN和其他机器学习系统的理论。我将专注于开发新的极限定理，描述当参数和/或数据的数量变得非常大时的行为。这些结果将帮助我们了解DNN的工作原理，并帮助我们在未来设计更有效的系统。我的研究计划的目标包括：1.神经切核：一个解释大型网络行为的随机矩阵。神经切核(NTK)是最近发现的一个非随机渐近对象，它解释了固定深度的DNN在无限宽度限制下的行为，当每个隐层中的神经元数量趋于无穷大时。当应用于随机数据时，NTK给出一个随机矩阵，其维度是给定数据点的数目。对这种随机矩阵的分析可以解释DNN在训练过程中的行为，并可以用来理解深度神经网络中的泛化误差。我建议用随机矩阵理论来研究这个模型。2.应用自由概率：用于随机矩阵分析的高级工具自由概率理论最初是针对算子代数领域的纯问题而发展起来的。然而，最近，来自自由概率及其推广的方法已经成为计算复杂随机矩阵模型的渐近特征的有力工具。一种应用是使用自由概率来计算连接到DNN的大型随机矩阵模型的极限谱。我还计划研究算子值自由概率的使用，它是自由概率的一个强大扩展，用于研究与DNN相关的块随机矩阵。3.Kardar-Parisi-Zhang(KPZ)普适性：随机矩阵本征值的涨落KPZ普适性类是随机系统的集合，包括随机偏微分方程组和相互作用的粒子系统的例子，它们都具有相同类型的普遍渐近随机行为。一个重要的应用是许多随机矩阵模型中最大特征值的行为。(与由其他随机矩阵工具捕获的频谱的整体行为相反)。我计划将KPZ中的思想应用于DNN和其他统计学习模型中的随机矩阵问题，以分析这些问题中最大特征值的演变。