Collaborative Research: Probabilistic, Geometric, and Topological Analysis of Neural Networks, From Theory to Applications

过去十年中最激动人心的技术发展之一是广泛采用了一系列名为神经网络的算法，这些算法被用于从自动驾驶汽车到根据蛋白质的氨基酸序列预测蛋白质的三维形状等尖端工业应用。这个项目的目标是双重的。首先，研究人员试图使用数学中的工具(特别是概率和组合学)来更好地理解神经网络的行为，然后将这种理解转化为新的、更有效和更安全的算法。这涉及到数学家、计算机科学家和电气工程师之间的合作努力。该项目团队试图解开一个根本的谜团：为什么神经网络看起来非常复杂，尽管它们看起来很复杂，但仍然学习节俭和有用的预测方法？换句话说，研究人员的目标是定义和分析神经网络复杂性的不同数学概念，然后将它们用作理论上的指导，以寻找与神经网络相关的更高效、更易解释的算法。第二个目标是创建一系列教育资源，从视频到课程笔记，使社会各阶层(如学生、政策制定者、科学家等)能够参与并对围绕现代神经网络的想法、挑战和机会进行有用的欣赏。本项目的研究由三个相互关联的部分组成。首先是对训练前、训练中和训练后的各种神经网络复杂性度量进行概率分析。相关工具来自概率论、泛函分析、信息论和几何学。关键的理论问题包括结构函数学习中隐含偏差的量化和泛化误差的定界。第二部分是对单个RELU网络函数和RELU网络空间的拓扑和几何分析。相关工具来自Morse理论和低维拓扑学。关键的理论问题在于对拓扑隐含偏差和拓扑深度分离的理解。最后，研究人员通过以下方式为应用深度学习寻求理论指导的见解：(I)原则性、高效的神经架构搜索，使用平均案例复杂性度量作为实际表现力、可训练性和泛化的替代品，以及(Ii)通过RELU网络的拓扑表现力来进行模型压缩和缩放的新方法。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

One of the most exciting technical developments of the last decade is the widespread adoption of a family of algorithms called neural networks, used in cutting-edge industrial applications ranging from self-driving cars to predicting the three-dimensional shapes of proteins from their amino acid sequences. The goals of this project are twofold. First, the investigators seek to use tools from mathematics (specifically probability and combinatorics) to better understand how neural networks behave and then to fashion this understanding into new, more efficient, and safer algorithms. This involves a collaborative effort between mathematicians, computer scientists, and electrical engineers. The project team seeks to unravel a fundamental mystery: why is it that neural networks appear to be incredibly complex, yet despite their seeing intricacy, still learn parsimonious and useful ways of making predictions? Put another way, the investigators aim to define and analyze different mathematical notions of neural network complexity and then to use them as theoretically grounded guides in the search for ever more efficient and interpretable algorithms related to neural networks. The second goal is to create a series of educational resources, ranging from videos to course notes, that will enable various segments of society at large (e.g. students, policy makers, scientists, and so on) to engage with and get a usable appreciation for the ideas, challenges, and opportunities surrounding modern neural networks. The research in this project consists of three interconnected parts. The first is a probabilistic analysis of a variety of neural network complexity measures before, during, and after training. Relevant tools come from probability, functional analysis, information theory, and geometry. Key theoretical questions include quantifying implicit bias and bounding generalization error for learning structured functions. The second is a topological and geometric analysis of both individual ReLU network functions and spaces of ReLU networks. Relevant tools come from Morse Theory and low-dimensional topology. Key theoretical questions hinge on understanding topological implicit bias and topological depth separation. Finally, the investigators seek theory-guided insights for applied deep learning via (i) principled, efficient neural architecture search using average case complexity measures as surrogates for practical expressivity, trainability, and generalization and (ii) novel approaches to model compression and scaling via topological expressivity of ReLU networks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.