Random Neural Networks

神经网络是在过去几年中在各种重要的机器学习任务中实现了最先进的算法，从计算机视觉(例如自动驾驶汽车)到自然语言处理(例如Echo、Alex、谷歌翻译等)和强化学习(例如AlphaGo和AlphaStar)。尽管取得了这些令人印象深刻的成功，但尚不清楚神经网络为什么工作得这么好。在这个项目中，PI将使用概率中的工具来发展我们对神经网络的理论理解。我们的目标是让我们深入理解为什么神经网络在克服优化和高维数据分析方面的挑战方面如此有效。这些理论见解将反过来为构建下一代基于神经网络的机器学习系统的工程师提供直觉。从数学上讲，神经网络的研究是近似论和最优化的交叉，涉及随机矩阵理论、高斯过程、超平面排列、张量分解和最优传输等主题。PI将特别关注(I)基于梯度的神经网络优化对于由许多随机矩阵的乘积给出的随机矩阵集合的线性统计和谱渐近的稳定性，在乘积中的项数和矩阵的大小同时成组的情况下，以及(Ii)在初始化时计算神经网络的相关函数(例如，使用随机权重和偏差)。类型(I)的问题给出了关于神经网络结构在初始化时的数值稳定性的定量信息。相比之下，类型(II)的问题旨在数据驱动的架构选择原则。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Neural networks are algorithms that in the past several years have achieved state of the art in a variety of important machine learning tasks, ranging from computer vision (e.g. self-driving cars) to natural language processing (e.g. Echo, Alex, Google Translate, etc) and reinforcement learning (e.g. AlphaGo and AlphaStar). Despite these impressive successes, it is not clear why neural nets work so well. In this project, the PI will use tools from probability to develop our theoretical understanding of neural networks. The goal is to give us a deep understanding of why neural nets are so efficient at overcoming challenges in optimization and high-dimensional data analysis. These theoretical insights will, in turn, inform the intuition of engineers for building the next generation of neural net-based machine learning systems. Mathematically, the study of neural networks is a cross between approximation theory and optimization, touching on topics from random matrix theory, Gaussian processes, hyperplane arrangements, tensor decompositions, and optimal transport, to name a few. The PI will focus specifically on (i) the stability of gradient-based optimization of neural networks to both the linear statistics and spectral asymptotics of random matrix ensembles given by products of many random matrices in the regime where both the number of terms in the product and the sizes of the matrices simultaneously group, and (ii) computing the correlation functions of neural networks at initialization (e.g. with random weights and biases). Questions of type (i) give quantitative information on the numerical stability of neural network architectures at initialization. Questions of type (ii), in contrast, aim at principles for data-driven architecture selection.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.