Random Matrix Limit Theorems for Deep Neural Networks

深度神经网络（dnn）的最新进展对现代世界产生了巨大的影响。然而，对这些系统的理论认识仍处于起步阶段。当然，这一领域的研究一直以经验为导向，以计算为重点，而不是强调数学上严谨的结果。有许多尚未解决的理论问题已经被实证工作发现，现在已经成熟，可以进行数学分析。我提出了一个研究计划，将开发和应用理论概率工具，特别是随机矩阵理论，以更好地理解深度神经网络和其他机器学习系统的理论。我将重点发展新的极限定理，描述参数和/或数据数量变得非常大时的行为。这些结果将帮助我们理解深度神经网络是如何工作的，并帮助我们在未来设计更有效的系统。我的研究计划的目标包括：1。神经切线核（neural tangent kernel， NTK）是最近发现的一种非随机渐近对象，它解释了当每个隐藏层中的神经元数量趋于无穷大时，固定深度dnn在无限宽度限制下的行为。当应用于随机数据时，NTK给出一个随机矩阵，其维度是给定数据点的数量。对该随机矩阵的分析可以解释dnn在训练过程中的行为，并可用于理解深度神经网络中的泛化误差。我建议用随机矩阵理论来研究这个模型。2. 应用自由概率：随机矩阵分析的先进工具自由概率理论最初是在算子代数领域的纯问题中发展起来的。然而，最近，自由概率及其扩展的方法已经成为计算复杂随机矩阵模型的渐近特征的强大工具。一个应用是使用自由概率来计算连接到深度神经网络的大型随机矩阵模型的极限谱。我还计划研究使用算子值自由概率（自由概率的强大扩展）来研究与dnn相关的块随机矩阵。3. KPZ通用性类是随机系统的集合，包括随机偏微分方程和相互作用粒子系统的例子，它们都具有相同类型的普遍渐近随机行为。一个重要的应用是在许多随机矩阵模型中最大特征值的行为。（与其他随机矩阵工具捕获的光谱的整体行为相反）。我计划将KPZ的思想应用到来自dnn和其他统计学习模型的随机矩阵问题中，分析这些问题中最大特征值的演化。

Recent advances in deep neural networks (DNNs) have had a tremendous impact on the modern world. However, the theoretical understanding of these systems is still in its infancy. As a matter of course, the research in this area has been empirically driven and computationally focused rather than emphasizing mathematically rigorous results. There are many open theoretical questions which have been uncovered by empirical work that are now ripe for mathematical analysis. I propose a research program that will develop and apply tools from theoretical probability, specifically random matrix theory, to gain a better understanding of the theory of DNNs and other machine learning systems. I will focus on developing new limit theorems which describe behavior when the number of parameters and/or data becomes very large. These results will help us understand how DNNs work and help us design more effective systems in the future. My objectives of the research program include: 1. The neural tangent kernel: A random matrix that explains the behavior of large networks The neural tangent kernel (NTK) is a recently discovered non-random asymptotic object that explains the behavior of DNNs of fixed depth in the infinite width limit, when the number of neurons in each hidden layer tends to infinity. When applied to random data, the NTK gives a random matrix whose dimensions are the number of given data points. Analysis of this random matrix can explain how DNNs behave during training and can be used to understand the generalization error in deep neural networks. I propose to study this model using random matrix theory. 2. Applied free probability: Advanced tools for random matrix analysis The theory of free probability was originally developed in connection to pure problems in the field of operator algebras. More recently however, methods from free probability and its extensions have emerged as powerful tools for computing asymptotic features of complicated random matrix models. One application is to use free probability to compute the limiting spectrum of large random matrix models connected to DNNs. I also plan to investigate the use of operator valued free probability, a powerful extension of free probability, to study block random matrices related to DNNs. 3. Kardar-Parisi-Zhang (KPZ) universality: Fluctuations of random matrix eigenvalues The KPZ universality class is a collection of stochastic systems, including examples from stochastic PDEs and interacting particle systems, which all share the same type of universal asymptotic random behavior. An important application is the behaviour of the largest eigenvalues in many random matrix models. (As opposed to the bulk behavior of the spectrum captured by other random matrix tools). I plan to apply ideas from KPZ to random matrix problems coming from DNNs and other statistical learning models to analyze the evolution of the largest eigenvalues in these problems.