Adaptive Neural Networks for Partial Differential Equations

神经网络在计算机视觉、自然语言处理和许多其他人工智能任务中取得了惊人的表现。尽管神经网络在许多实际应用中取得了巨大的成功，但人们普遍认为神经网络的近似性质尚未得到很好的理解。这个项目将有可能理解神经网络为什么以及如何工作，特别是，为什么神经网络比其他函数类更好地用于具有奇点和不连续问题的计算机模拟。在这个项目中开发的自适应神经网络方法将有潜力极大地提高计算机模拟复杂物理、生物和人类工程系统的计算能力，这些系统表现出计算困难。对神经网络宽度和深度在近似中的作用的理解，以及在本项目中获得的新思想，将对许多其他人工智能任务，如迁移学习、在线控制和模式识别产生重大影响。期望至少有一名毕业生就所建议的研究课题进行培训。机器学习中一个基本的、开放的问题是如何设计神经网络的架构，就其宽度和深度而言，以便准确有效地近似函数或数值解偏微分方程。为应用程序解决计算上具有挑战性的问题是本项目的重点。更准确地说，对于任何给定的偏微分方程，该项目将开发一种自适应神经元增强（ANE）方法，该方法自适应地构建具有几乎最小数量的神经元和参数的神经网络，使其近似精度在规定的公差范围内。开发ANE方法的一个关键组成部分是一个错误指示器，用于确定在当前层或下一层添加的新神经元的数量。该项目计划通过研究当前层和下一层中额外神经元的作用来开发有效的指标。这些知识对于设计高效和准确的ANE方法至关重要，但通常不能由标准的先验误差估计提供。神经网络卓越的近似能力是有代价的：即使潜在的偏微分方程是线性的，确定参数值的过程现在也是一个非线性优化问题。非线性优化通常有许多解，只有从一个足够接近的第一近似开始才能得到期望的解。在这个项目中开发的ANE方法是一个很好的延续过程。本项目将重点研究如何在ANE方法的每个自适应阶段初始化新神经元的参数。这将通过分析每一层神经元的物理意义来完成。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Neural networks have achieved astonishing performance in computer vision, natural language processing, and many other artificial intelligence tasks. Despite their great successes in many practical applications, it is widely accepted that approximation properties of neural networks are not yet well-understood. This project would have the potential to understand why and how neural networks work, particularly, why a neural network is much better than other functional classes for computer simulations of problems with singularities and discontinuities. The self-adaptive neural network method developed in this project would have the potential to dramatically improve the computational capabilities for computer simulations of complex physical, biological, and human-engineered systems exhibiting computational difficulties. Understanding of the role of neural network width and depth in approximation and the new ideas gained in this project would have significant impacts on many other artificial intelligence tasks such as transfer learning, online control, and pattern recognition. Training of at least one graduate on the topics of the proposed research is expected. A fundamental, open question in machine learning is on how to design the architecture of neural networks, in terms of their width and depth, in order to approximate functions or numerically solve partial differential equations accurately and efficiently. Addressing this issue for applications to computationally challenging problems is the focus of this project. More precisely, for any given partial differential equation, this project is going to develop an adaptive neuron enhancement (ANE) method that adaptively constructs a neural network with a nearly minimum number of neurons and parameters such that its approximation accuracy is within the prescribed tolerance. A key component of developing ANE methods is an error indicator for determining the number of new neurons to be added at either the current or next layer. This project plans to develop efficient indicators by studying the role of additional neurons in the current and next layers. This knowledge is crucial for designing efficient and accurate ANE methods, but is generally not provided by the standard a priori error estimates. The exceptional approximation powers of neural networks come with a price: the procedure for determining the values of the parameters is now a problem in nonlinear optimization even if the underlying partial differential equation is linear. Nonlinear optimizations usually have many solutions, and the desired one is obtained only if one starts from a close enough first approximation. The ANE method to be developed in this project is a good continuation process. This project will focus on how to initialize parameters of new neurons at each adaptive stage of the ANE method. This will be done by analyzing the physical meanings of neurons at each layer.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.