Collaborative Research: CIF: Medium: Analysis and Geometry of Neural Dynamical Systems

现代神经网络的复杂性及其数百万个参数和前所未有的计算需求，一直是过去几十年来成功应用于机器学习的传统方法的主要障碍。该项目旨在通过将这些系统的架构作为连续动力系统进行全新的概念化，为这些系统的分析和设计开发新的数学和计算基础。这个框架的关键支柱是将深度理想化为连续的层，将宽度理想化为连续的神经元。这种类型的无穷小抽象在整个二十世纪成功地解开了许多学科，包括概率、优化、控制等等。这个涉及UIUC和麻省理工学院的合作项目将推动深度学习理论和实践的界限，同时引发电气工程，数学，统计和理论计算机科学社区之间的持续互动。该项目还将通过审慎的教育和培训办法产生广泛影响。教育和推广活动将包括为两个机构的本科生提供研究机会，以及促进合作和思想交流的交流计划。这个关于神经动力系统的分析和几何的项目正在通过综合概率，统计，动力系统，几何分析，偏微分方程和最佳运输的工具来开发深度学习的数学基础。研究计划围绕三个主要方向：（1）神经动力系统的连续模型;（2）离散化方案;（3）算法。第一个方向是集中在描述理想化的无限宽和无限深神经网络的表达能力和复杂性之间的权衡。第二个方向建立在这些连续抽象的基础上，从第一原理出发，开发数学上严格且实际上可实现的技术，用于分析大型但有限的神经网络。第三个方向强调算法和计算方面，例如数值方法的计算复杂性、稳定性和隐式正则化，使用作为项目一部分开发的分析和几何方法的新颖综合。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

The complexity of modern neural nets, with their millions of parameters and unprecedented computational demands, has been a major hurdle for the conventional approaches which had been successfully applied in machine learning over the past decades. This project aims to develop new mathematical and computational foundations for the analysis and design of these systems through a radically new conceptualization of their architectures as continuous dynamical systems. The key pillar of this framework is the idealization of depth as a continuum of layers and width as a continuum of neurons. Infinitesimal abstractions of this type have successfully unlocked many disciplines throughout the twentieth century, including probability, optimization, control, and many more. This collaborative project involving UIUC and MIT will push the boundaries of the theory and practice of deep learning, while sparking sustained interactions between the communities of electrical engineering, mathematics, statistics, and theoretical computer science. The project will also have broad impacts through a deliberate approach to education and training. The education and outreach activities will include research opportunities for undergraduate students at both institutions, as well as an exchange program to foster the collaboration and exchange of ideas. This project on Analysis and Geometry of Neural Dynamical Systems is developing the mathematical foundations of deep learning by synthesizing tools from probability, statistics, dynamical systems, geometric analysis, partial differential equations, and optimal transport. The research program is articulated around three major directions: (1) continuous models of neural dynamical systems; (2) discretization schemes; and (3) algorithms. The first direction is focusing on characterizing the tradeoffs between the expressive power and complexity of idealized infinitely wide and deep neural nets. The second direction builds on these continuous abstractions to develop, from first principles, mathematically rigorous and practically implementable techniques for analyzing large but finite neural nets. The third direction emphasizes algorithmic and computational aspects, such as the computational complexity of numerical methods, stability, and implicit regularization, using a novel synthesis of analytic and geometric methods developed as part of the project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.