Random Matrix Limit Theorems for Deep Neural Networks

点击翻译按钮获取中文摘要

Recent advances in deep neural networks (DNNs) have had a tremendous impact on the modern world. However, the theoretical understanding of these systems is still in its infancy. As a matter of course, the research in this area has been empirically driven and computationally focused rather than emphasizing mathematically rigorous results. There are many open theoretical questions which have been uncovered by empirical work that are now ripe for mathematical analysis. I propose a research program that will develop and apply tools from theoretical probability, specifically random matrix theory, to gain a better understanding of the theory of DNNs and other machine learning systems. I will focus on developing new limit theorems which describe behavior when the number of parameters and/or data becomes very large. These results will help us understand how DNNs work and help us design more effective systems in the future. My objectives of the research program include: 1. The neural tangent kernel: A random matrix that explains the behavior of large networks The neural tangent kernel (NTK) is a recently discovered non-random asymptotic object that explains the behavior of DNNs of fixed depth in the infinite width limit, when the number of neurons in each hidden layer tends to infinity. When applied to random data, the NTK gives a random matrix whose dimensions are the number of given data points. Analysis of this random matrix can explain how DNNs behave during training and can be used to understand the generalization error in deep neural networks. I propose to study this model using random matrix theory. 2. Applied free probability: Advanced tools for random matrix analysis The theory of free probability was originally developed in connection to pure problems in the field of operator algebras. More recently however, methods from free probability and its extensions have emerged as powerful tools for computing asymptotic features of complicated random matrix models. One application is to use free probability to compute the limiting spectrum of large random matrix models connected to DNNs. I also plan to investigate the use of operator valued free probability, a powerful extension of free probability, to study block random matrices related to DNNs. 3. Kardar-Parisi-Zhang (KPZ) universality: Fluctuations of random matrix eigenvalues The KPZ universality class is a collection of stochastic systems, including examples from stochastic PDEs and interacting particle systems, which all share the same type of universal asymptotic random behavior. An important application is the behaviour of the largest eigenvalues in many random matrix models. (As opposed to the bulk behavior of the spectrum captured by other random matrix tools). I plan to apply ideas from KPZ to random matrix problems coming from DNNs and other statistical learning models to analyze the evolution of the largest eigenvalues in these problems.