Collaborative Research: RI: Medium: MoDL: Occams Razor in Deep and Physical Learning

深度神经网络(DNN)是一种机器学习模型，灵感来自于神经元在动物大脑中执行计算的方式。在过去的十年里，这些模型在科学和工程的许多领域引发了革命，从预测手机键盘上的下一个单词，到在最能解释宇宙结构的宇宙学模型中进行选择。尽管计算机科学家已经在构建这些系统方面获得了专业知识，但他们目前还不知道它们为什么会工作，什么时候会失败。研究议程的重点是开发理论工具，这些工具将为DNN建立这样一个必要的理解，希望这些工具也将阐明学习是如何在生物系统中发生的，例如大脑中的神经元网络。该项目的智力目标是以人工和生物系统学习的方式确定共同的主题。教育和推广目标包括(A)开发计算机科学、神经科学和数学的交叉课程，(B)为费城的高中生组织人工智能教程，以及(C)指导Latinx数学研究社区的年轻研究人员。训练深层网络归结为高维、大规模和非凸优化问题；奇怪的是，像随机梯度下降这样的简单算法不仅足以训练DNN，而且看起来也是必要的。公认的统计学智慧表明，模型班级越大，学习的模型就越有可能超过训练数据。然而，DNN对新数据的泛化非常好。这个项目试图解开这一明显的悖论：中心假设是，当学习任务表现出一种称为“草率”的特征结构时，DNN成功。对于草率的学习任务，学习网络的Fisher信息矩阵的特征值均匀分布在矩阵的指数大小的范围内。这个项目将调查这种草率的结构如何导致训练过程只探索函数空间的一小部分，从而产生快速训练和良好的泛化。它将描述这个微小的子集的形状，以理解为什么网络在典型的学习任务中学习简单的低维函数。将与生物和物理系统建立联系，这些系统通过局部学习规则学习，并且也显示出这种松散的结构(例如，大脑中的神经元网络和弹性聚合物网络，如蛋白质)。技术目标是揭示学习的普遍原则，即DNN和物理学习网络所表现出的简单性和低维度的内部表现。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Deep neural networks (DNNs) are machine learning models inspired by how neurons perform computations in the animal brain. Over the past decade these models have led to revolutions in many fields of science and engineering, from making predictions of the next word on the keyboard of a mobile phone to selecting between cosmological models that best explain the structure of the universe. Although computer scientists have gained expertise in building these systems, they do not currently understand why they work and when they can fail. The research agenda focuses on developing theoretical tools that will build such a needed understanding for DNNs, with the hope that these same tools will also shed light on how learning occurs in biological systems, e.g., networks of neurons in the brain. The intellectual goal of the project is to identify common themes in the ways artificial and biological systems learn. The educational and outreach goals include (a) developing curricula at the intersection of computer science, neuroscience, and mathematics, (b) organizing tutorials on artificial intelligence for high-school students in Philadelphia, and (c) mentoring young researchers in the LatinX mathematical research community.Training a deep network reduces to a high-dimensional, large-scale, and non-convex optimization problem; curiously enough, simple algorithms like stochastic gradient descent are not just sufficient but also seemingly necessary for training DNNs. Accepted statistical wisdom suggests that the larger the model class, the more likely the learned model will overfit the training data. Yet, DNNs generalize extremely well to new data. This project seeks to unravel this apparent paradox: The central hypothesis is that DNNs succeed when the learning tasks exhibit a characteristic structure called “sloppiness.” For sloppy learning tasks, the Fisher Information Matrix of the learned network has eigenvalues that are distributed uniformly across a range that is exponentially large in the rank of the matrix. This project will investigate how this sloppy structure results in the training process exploring only a tiny subset of the function space, thereby yielding both rapid training and good generalization. It will characterize the shape of this tiny subset to understand why networks learn simple, low-dimensional functions for typical learning tasks. Connections will be made to biological and physical systems that learn through local learning rules and also exhibit such a sloppy structure (e.g., networks of neurons in the brain and elastic polymer networks such as proteins). The technical objective is to reveal universal principles of learning, namely a drive towards simplicity and low-dimensional internal representations exhibited by both DNNs and physical learning networks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.