Random Neural Networks

随机神经网络

• 批准号: 2045167
• 负责人: Boris Hanin
• 金额: $ 14.28万
• 依托单位: Princeton University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-08-01 至 2023-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2045167&HistoricalAwards=false
• 关键词: Random; Neural; Networks

Neural networks are algorithms that in the past several years have achieved state of the art in a variety of important machine learning tasks, ranging from computer vision (e.g. self-driving cars) to natural language processing (e.g. Echo, Alex, Google Translate, etc) and reinforcement learning (e.g. AlphaGo and AlphaStar). Despite these impressive successes, it is not clear why neural nets work so well. In this project, the PI will use tools from probability to develop our theoretical understanding of neural networks. The goal is to give us a deep understanding of why neural nets are so efficient at overcoming challenges in optimization and high-dimensional data analysis. These theoretical insights will, in turn, inform the intuition of engineers for building the next generation of neural net-based machine learning systems. Mathematically, the study of neural networks is a cross between approximation theory and optimization, touching on topics from random matrix theory, Gaussian processes, hyperplane arrangements, tensor decompositions, and optimal transport, to name a few. The PI will focus specifically on (i) the stability of gradient-based optimization of neural networks to both the linear statistics and spectral asymptotics of random matrix ensembles given by products of many random matrices in the regime where both the number of terms in the product and the sizes of the matrices simultaneously group, and (ii) computing the correlation functions of neural networks at initialization (e.g. with random weights and biases). Questions of type (i) give quantitative information on the numerical stability of neural network architectures at initialization. Questions of type (ii), in contrast, aim at principles for data-driven architecture selection.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

神经网络是在过去几年中在各种重要的机器学习任务中达到最先进水平的算法，从计算机视觉（例如自动驾驶汽车）到自然语言处理（例如Echo，Alex，Google翻译等）和强化学习（例如AlphaGo和AlphaStar）。尽管取得了这些令人印象深刻的成功，但尚不清楚为什么神经网络工作得如此出色。在这个项目中，PI将使用概率工具来发展我们对神经网络的理论理解。我们的目标是让我们深入了解为什么神经网络在克服优化和高维数据分析方面的挑战时如此有效。反过来，这些理论见解将为工程师构建下一代基于神经网络的机器学习系统的直觉提供信息。在数学上，神经网络的研究是近似理论和优化之间的交叉，涉及随机矩阵理论，高斯过程，超平面排列，张量分解和最优传输等主题。PI将特别关注（i）神经网络的基于梯度的优化对由许多随机矩阵的乘积给出的随机矩阵集合的线性统计和谱渐近的稳定性，其中乘积中的项数和矩阵的大小同时分组，以及（ii）在初始化时计算神经网络的相关函数（例如，具有随机权重和偏置）。类型（i）的问题给出了关于初始化时神经网络结构的数值稳定性的定量信息。与此相反，类型（ii）的问题针对的是数据驱动架构选择的原则。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Deep ReLU Networks Have Surprisingly Few Activation Patterns

深度 ReLU 网络的激活模式少得惊人

• 发表时间: 2019
• 期刊: Advances in neural information processing systems
• 作者: Hanin, Boris;Rolnick, David
• 通讯作者: Rolnick, David

Complexity of Linear Regions in Deep Networks

• 发表时间: 2019-01
• 作者: B. Hanin;D. Rolnick

How Data Augmentation affects Optimization for Linear Regression

• 发表时间: 2020-10
• 作者: B. Hanin;Yi Sun

Deep ReLU networks preserve expected length

深度 ReLU 网络保留预期长度

• 发表时间: 2022
• 期刊: ICLR
• 作者: Hanin, B.;Jeong, R.;Rolnick, D.
• 通讯作者: Rolnick, D.

Finite Depth and Width Corrections to the Neural Tangent Kernel

• 发表时间: 2019-09
• 期刊: ArXiv
• 作者: B. Hanin;M. Nica