CAREER: Random Neural Nets and Random Matrix Products

我们生活在一个大数据和廉价计算的时代。机器学习算法可以有效地分析海量信息的潜在模式，导致从自动驾驶汽车到自动药物发现和机器翻译等应用领域的显著进展。支撑许多这些令人兴奋的实际发展的是一类被称为神经网络的计算模型。最初发展于1940年的S和1950年的S，今天使用的神经网络既复杂又强大。该项目的目的是开发一系列原则性技术，以了解神经网络在实践中如何工作以及如何使其更好的关键方面。本项目采取的方法是概率和统计性质的。就像理想气体定律直接通过压力、体积和温度准确地描述气体的大尺度性质一样，这个项目将探索和识别大型神经网络的紧急统计行为，这些行为可以证明它们在实践中观察到的许多关键性质。该项目还将通过为研究生组织机器学习暑期学校，提供研究、培训和教育机会。在更高的层次上，神经网络是通过将仿射变换与基本的非线性运算组合在一起而给出的一族函数。最简单、最重要的一类神经网络大致可以用深度和宽度这两个参数来描述。前者是仿射变换作用的空间的维度，后者是合成的数目。该项目的技术核心是了解随机选择仿射变换时此类网络的统计行为。起始点是一种分析上容易处理的制度，在这种制度下，网络宽度在固定深度被发送到无穷大。在这个无限宽度的限制下，随机网络收敛到高斯过程，从随机选择的起点优化网络参数就变成了一种核方法。不幸的是，这种简明的描述不能捕捉到神经网络最重要的经验特性，即它们学习依赖于数据的特征的能力。了解特征学习是如何发生的是这个项目的核心，需要新的概率和分析工具来研究有限宽度的随机神经网络。其基本思想是围绕无限大的宽度极限进行摄动理论，将网络宽度的倒数视为一个小参数。然后，目标是获得随机神经网络的值和导数(相对于模型输入和模型参数)的联合分布的表达式，对于该倒数中的所有阶。这些公式对于理解神经网络训练的数值稳定性，建议优化超参数的原则性设置，以及量化特征学习具有实际意义。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

We live in an era of big data and inexpensive computation. Vast stores of information can efficiently be analyzed for underlying patterns by machine learning algorithms, leading to remarkable progress in applications ranging from self-driving cars to automatic drug discovery and machine translation. Underpinning many of these exciting practical developments is a class of computational models called neural networks. Originally developed in the 1940's and 1950's, the neural nets used today are as complex as they are powerful. The purpose of this project is to develop a range of principled techniques for understanding key aspects of how neural networks work in practice and how to make them better. The approach taken by this project is probabilistic and statistical in nature. Just as the ideal gas law accurately describes the large-scale properties of a gas directly through pressure, volume, and temperature without the need specify the state of each individual gas molecule, this project will explore and identify emergent statistical behaviors of large neural networks that provably explain many of their key properties observed in practice. The project will also provide research training and educational opportunities through organization of summer schools in machine learning for graduate students. At a high level, a neural network is a family of functions given by composing affine transformations with elementary non-linear operations. The simplest important kind of neural networks are roughly described by two parameters called depth and width. The former is the dimension of the spaces on which the affine transformations act and the latter is the number of compositions. The technical heart of this project is to understand the statistical behavior of such networks when the affine transformations are chosen at random. The starting point is an analytically tractable regime in which the network width is sent to infinity at fixed depth. In this infinite width limit, random networks converge to Gaussian processes and optimization of network parameters from their randomly chosen starting points reduces to a kernel method. Unfortunately, this concise description cannot capture what is perhaps the most important empirical property of neural networks, namely their ability to learn data-dependent features. Understanding how feature learning occurs is at the core of this project and requires new probabilistic and analytic tools for studying random neural networks at finite width. The basic idea is to perform perturbation theory around the infinite width limit, treating the reciprocal of the network width as a small parameter. The goal is then to obtain, to all orders in this reciprocal, the expressions for joint distribution of the values and derivatives (with respect to both model inputs and model parameters) of a random neural network. Such formulas have practical consequences for understanding the numerical stability of neural network training, suggesting principled settings for optimization hyper-parameters, and quantifying feature learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.