Random Neural Networks

神经网络是过去几年在各种重要机器学习任务中实现最先进的算法，从计算机视觉（例如自动驾驶汽车）到自然语言处理（例如 Echo、Alex、Google Translate 等）和强化学习（例如 AlphaGo 和 AlphaStar）。尽管取得了这些令人印象深刻的成功，但尚不清楚神经网络为何如此有效。在这个项目中，PI 将使用概率工具来发展我们对神经网络的理论理解。目的是让我们深入了解为什么神经网络能够如此有效地克服优化和高维数据分析中的挑战。这些理论见解反过来将为工程师构建下一代基于神经网络的机器学习系统提供直觉。从数学上讲，神经网络的研究是近似理论和优化之间的交叉，涉及随机矩阵理论、高斯过程、超平面排列、张量分解和最优传输等主题。 PI 将特别关注 (i) 基于梯度的神经网络优化对随机矩阵系综的线性统计和谱渐近的稳定性，随机矩阵系综是由乘积中的项数和矩阵大小同时分组的情况下的许多随机矩阵的乘积给出的，以及 (ii) 在初始化时计算神经网络的相关函数（例如，使用随机权重和偏差）。 (i) 类型的问题给出了神经网络架构在初始化时的数值稳定性的定量信息。相反，类型 (ii) 的问题针对数据驱动架构选择的原则。该奖项反映了 NSF 的法定使命，并且通过使用基金会的智力优点和更广泛的影响审查标准进行评估，被认为值得支持。

Neural networks are algorithms that in the past several years have achieved state of the art in a variety of important machine learning tasks, ranging from computer vision (e.g. self-driving cars) to natural language processing (e.g. Echo, Alex, Google Translate, etc) and reinforcement learning (e.g. AlphaGo and AlphaStar). Despite these impressive successes, it is not clear why neural nets work so well. In this project, the PI will use tools from probability to develop our theoretical understanding of neural networks. The goal is to give us a deep understanding of why neural nets are so efficient at overcoming challenges in optimization and high-dimensional data analysis. These theoretical insights will, in turn, inform the intuition of engineers for building the next generation of neural net-based machine learning systems. Mathematically, the study of neural networks is a cross between approximation theory and optimization, touching on topics from random matrix theory, Gaussian processes, hyperplane arrangements, tensor decompositions, and optimal transport, to name a few. The PI will focus specifically on (i) the stability of gradient-based optimization of neural networks to both the linear statistics and spectral asymptotics of random matrix ensembles given by products of many random matrices in the regime where both the number of terms in the product and the sizes of the matrices simultaneously group, and (ii) computing the correlation functions of neural networks at initialization (e.g. with random weights and biases). Questions of type (i) give quantitative information on the numerical stability of neural network architectures at initialization. Questions of type (ii), in contrast, aim at principles for data-driven architecture selection.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.