Adaptive Neural Networks for Partial Differential Equations

神经网络在计算机视觉、自然语言处理和许多其他人工智能任务中取得了惊人的性能。尽管它们在许多实际应用中取得了巨大的成功，但人们普遍认为，神经网络的逼近性质还没有被很好地理解。这个项目将有可能理解神经网络为什么以及如何工作，特别是为什么神经网络在计算机模拟具有奇异性和不连续性的问题时比其他功能类要好得多。本项目中开发的自适应神经网络方法将有可能极大地提高计算机模拟复杂的物理、生物和人类工程系统的计算能力。对神经网络宽度和深度在近似中的作用的理解以及在这个项目中获得的新想法将对许多其他人工智能任务产生重大影响，如转移学习、在线控制和模式识别。预计将至少对一名毕业生进行关于拟议研究主题的培训。机器学习中一个基本的、尚待解决的问题是如何根据神经网络的宽度和深度设计神经网络的结构，以便准确有效地逼近函数或数值求解偏微分方程组。解决这一问题的应用程序的计算挑战问题是本项目的重点。更准确地说，对于任何给定的偏微分方程，本项目将开发一种自适应神经元增强(ANE)方法，该方法以几乎最少的神经元和参数自适应地构建神经网络，使其逼近精度在规定的容差内。开发ANE方法的一个关键组成部分是用于确定要在当前层或下一层添加的新神经元数量的错误指示器。该项目计划通过研究当前和下一层中额外神经元的作用来开发有效的指标。这一知识对于设计有效和准确的ANE方法是至关重要的，但通常不是标准先验误差估计所提供的。神经网络非凡的逼近能力是有代价的：即使基本的偏微分方程式是线性的，确定参数值的过程现在也是非线性优化中的一个问题。非线性优化通常有许多解，只有当一个人从足够接近的第一近似值开始时，才能获得所需的解。这个项目中将要开发的一个方法是一个很好的延续过程。这个项目的重点是如何在ANE方法的每个自适应阶段初始化新神经元的参数。这将通过分析每一层神经元的物理意义来完成。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Neural networks have achieved astonishing performance in computer vision, natural language processing, and many other artificial intelligence tasks. Despite their great successes in many practical applications, it is widely accepted that approximation properties of neural networks are not yet well-understood. This project would have the potential to understand why and how neural networks work, particularly, why a neural network is much better than other functional classes for computer simulations of problems with singularities and discontinuities. The self-adaptive neural network method developed in this project would have the potential to dramatically improve the computational capabilities for computer simulations of complex physical, biological, and human-engineered systems exhibiting computational difficulties. Understanding of the role of neural network width and depth in approximation and the new ideas gained in this project would have significant impacts on many other artificial intelligence tasks such as transfer learning, online control, and pattern recognition. Training of at least one graduate on the topics of the proposed research is expected. A fundamental, open question in machine learning is on how to design the architecture of neural networks, in terms of their width and depth, in order to approximate functions or numerically solve partial differential equations accurately and efficiently. Addressing this issue for applications to computationally challenging problems is the focus of this project. More precisely, for any given partial differential equation, this project is going to develop an adaptive neuron enhancement (ANE) method that adaptively constructs a neural network with a nearly minimum number of neurons and parameters such that its approximation accuracy is within the prescribed tolerance. A key component of developing ANE methods is an error indicator for determining the number of new neurons to be added at either the current or next layer. This project plans to develop efficient indicators by studying the role of additional neurons in the current and next layers. This knowledge is crucial for designing efficient and accurate ANE methods, but is generally not provided by the standard a priori error estimates. The exceptional approximation powers of neural networks come with a price: the procedure for determining the values of the parameters is now a problem in nonlinear optimization even if the underlying partial differential equation is linear. Nonlinear optimizations usually have many solutions, and the desired one is obtained only if one starts from a close enough first approximation. The ANE method to be developed in this project is a good continuation process. This project will focus on how to initialize parameters of new neurons at each adaptive stage of the ANE method. This will be done by analyzing the physical meanings of neurons at each layer.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.