Collaborative Research: Probabilistic, Geometric, and Topological Analysis of Neural Networks, From Theory to Applications

过去十年中最令人兴奋的技术发展之一是被称为神经网络的一系列算法的广泛采用，这些算法被用于从自动驾驶汽车到根据氨基酸序列预测蛋白质三维形状等尖端工业应用中。这个项目有两个目标。首先，研究人员试图使用数学工具（特别是概率和组合学）来更好地理解神经网络的行为，然后将这种理解塑造成新的、更有效的、更安全的算法。这需要数学家、计算机科学家和电气工程师的共同努力。该项目团队试图解开一个基本的谜团：为什么神经网络看起来异常复杂，但尽管它们看到了复杂性，仍然学会了简单而有用的预测方法？换句话说，研究人员的目标是定义和分析神经网络复杂性的不同数学概念，然后将它们作为理论基础指导，以寻找与神经网络相关的更有效和可解释的算法。第二个目标是创建一系列教育资源，从视频到课程笔记，这将使社会各阶层（例如学生，政策制定者，科学家等）能够参与并获得有关现代神经网络的想法，挑战和机会的可用评价。本项目的研究由三个相互关联的部分组成。首先是对训练前、训练中和训练后的各种神经网络复杂性度量进行概率分析。相关工具来自概率论、功能分析、信息论和几何学。关键的理论问题包括量化内隐偏差和边界泛化误差的学习结构函数。其次，对ReLU网络的各个函数和空间进行了拓扑和几何分析。相关工具来自莫尔斯理论和低维拓扑。关键的理论问题取决于对拓扑内隐偏差和拓扑深度分离的理解。最后，研究人员通过(i)使用平均案例复杂性度量作为实际表达性、可训练性和泛化的替代方法进行原则性、高效的神经架构搜索，以及（ii）通过ReLU网络的拓扑表达性来进行模型压缩和缩放的新方法，寻求理论指导的深度学习应用见解。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

One of the most exciting technical developments of the last decade is the widespread adoption of a family of algorithms called neural networks, used in cutting-edge industrial applications ranging from self-driving cars to predicting the three-dimensional shapes of proteins from their amino acid sequences. The goals of this project are twofold. First, the investigators seek to use tools from mathematics (specifically probability and combinatorics) to better understand how neural networks behave and then to fashion this understanding into new, more efficient, and safer algorithms. This involves a collaborative effort between mathematicians, computer scientists, and electrical engineers. The project team seeks to unravel a fundamental mystery: why is it that neural networks appear to be incredibly complex, yet despite their seeing intricacy, still learn parsimonious and useful ways of making predictions? Put another way, the investigators aim to define and analyze different mathematical notions of neural network complexity and then to use them as theoretically grounded guides in the search for ever more efficient and interpretable algorithms related to neural networks. The second goal is to create a series of educational resources, ranging from videos to course notes, that will enable various segments of society at large (e.g. students, policy makers, scientists, and so on) to engage with and get a usable appreciation for the ideas, challenges, and opportunities surrounding modern neural networks. The research in this project consists of three interconnected parts. The first is a probabilistic analysis of a variety of neural network complexity measures before, during, and after training. Relevant tools come from probability, functional analysis, information theory, and geometry. Key theoretical questions include quantifying implicit bias and bounding generalization error for learning structured functions. The second is a topological and geometric analysis of both individual ReLU network functions and spaces of ReLU networks. Relevant tools come from Morse Theory and low-dimensional topology. Key theoretical questions hinge on understanding topological implicit bias and topological depth separation. Finally, the investigators seek theory-guided insights for applied deep learning via (i) principled, efficient neural architecture search using average case complexity measures as surrogates for practical expressivity, trainability, and generalization and (ii) novel approaches to model compression and scaling via topological expressivity of ReLU networks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.