Collaborative Research: CIF: Medium: Analysis and Geometry of Neural Dynamical Systems

现代神经网络的复杂性，以及数以百万计的参数和前所未有的计算需求，已经成为过去几十年来成功应用于机器学习的传统方法的主要障碍。该项目旨在通过将其架构作为连续动力系统的全新概念，为这些系统的分析和设计开发新的数学和计算基础。该框架的关键支柱是将深度理想化为层的连续体，将宽度理想化为神经元的连续体。这种类型的无限小抽象成功地开启了整个20世纪的许多学科，包括概率、优化、控制等等。这个由UIUC和MIT参与的合作项目将推动深度学习理论和实践的边界，同时激发电气工程、数学、统计学和理论计算机科学社区之间的持续互动。该项目还将通过审慎的教育和培训办法产生广泛的影响。教育和推广活动将包括为两所大学的本科生提供研究机会，以及促进合作和思想交流的交流项目。这个关于神经动力系统的分析和几何的项目是通过综合概率、统计、动力系统、几何分析、偏微分方程和最优运输等工具来发展深度学习的数学基础。研究方向围绕三个主要方向：(1)神经动力系统的连续模型；(2)离散化方案；(3)算法。第一个方向侧重于描述理想化的无限宽和深度神经网络的表达能力和复杂性之间的权衡。第二个方向是建立在这些连续抽象的基础上，从第一原理出发，发展数学上严谨和实际可实现的技术，用于分析大型但有限的神经网络。第三个方向强调算法和计算方面，例如数值方法的计算复杂性，稳定性和隐式正则化，使用作为项目一部分开发的解析和几何方法的新颖综合。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The complexity of modern neural nets, with their millions of parameters and unprecedented computational demands, has been a major hurdle for the conventional approaches which had been successfully applied in machine learning over the past decades. This project aims to develop new mathematical and computational foundations for the analysis and design of these systems through a radically new conceptualization of their architectures as continuous dynamical systems. The key pillar of this framework is the idealization of depth as a continuum of layers and width as a continuum of neurons. Infinitesimal abstractions of this type have successfully unlocked many disciplines throughout the twentieth century, including probability, optimization, control, and many more. This collaborative project involving UIUC and MIT will push the boundaries of the theory and practice of deep learning, while sparking sustained interactions between the communities of electrical engineering, mathematics, statistics, and theoretical computer science. The project will also have broad impacts through a deliberate approach to education and training. The education and outreach activities will include research opportunities for undergraduate students at both institutions, as well as an exchange program to foster the collaboration and exchange of ideas. This project on Analysis and Geometry of Neural Dynamical Systems is developing the mathematical foundations of deep learning by synthesizing tools from probability, statistics, dynamical systems, geometric analysis, partial differential equations, and optimal transport. The research program is articulated around three major directions: (1) continuous models of neural dynamical systems; (2) discretization schemes; and (3) algorithms. The first direction is focusing on characterizing the tradeoffs between the expressive power and complexity of idealized infinitely wide and deep neural nets. The second direction builds on these continuous abstractions to develop, from first principles, mathematically rigorous and practically implementable techniques for analyzing large but finite neural nets. The third direction emphasizes algorithmic and computational aspects, such as the computational complexity of numerical methods, stability, and implicit regularization, using a novel synthesis of analytic and geometric methods developed as part of the project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.