Adaptive Neural Networks for Partial Differential Equations

偏微分方程的自适应神经网络

• 批准号: 2110571
• 负责人: Zhiqiang Cai
• 金额: $ 33万
• 依托单位: Purdue University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-06-01 至 2024-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2110571&HistoricalAwards=false
• 关键词: Adaptive; Neural; Networks; Partial; Differential

Neural networks have achieved astonishing performance in computer vision, natural language processing, and many other artificial intelligence tasks. Despite their great successes in many practical applications, it is widely accepted that approximation properties of neural networks are not yet well-understood. This project would have the potential to understand why and how neural networks work, particularly, why a neural network is much better than other functional classes for computer simulations of problems with singularities and discontinuities. The self-adaptive neural network method developed in this project would have the potential to dramatically improve the computational capabilities for computer simulations of complex physical, biological, and human-engineered systems exhibiting computational difficulties. Understanding of the role of neural network width and depth in approximation and the new ideas gained in this project would have significant impacts on many other artificial intelligence tasks such as transfer learning, online control, and pattern recognition. Training of at least one graduate on the topics of the proposed research is expected. A fundamental, open question in machine learning is on how to design the architecture of neural networks, in terms of their width and depth, in order to approximate functions or numerically solve partial differential equations accurately and efficiently. Addressing this issue for applications to computationally challenging problems is the focus of this project. More precisely, for any given partial differential equation, this project is going to develop an adaptive neuron enhancement (ANE) method that adaptively constructs a neural network with a nearly minimum number of neurons and parameters such that its approximation accuracy is within the prescribed tolerance. A key component of developing ANE methods is an error indicator for determining the number of new neurons to be added at either the current or next layer. This project plans to develop efficient indicators by studying the role of additional neurons in the current and next layers. This knowledge is crucial for designing efficient and accurate ANE methods, but is generally not provided by the standard a priori error estimates. The exceptional approximation powers of neural networks come with a price: the procedure for determining the values of the parameters is now a problem in nonlinear optimization even if the underlying partial differential equation is linear. Nonlinear optimizations usually have many solutions, and the desired one is obtained only if one starts from a close enough first approximation. The ANE method to be developed in this project is a good continuation process. This project will focus on how to initialize parameters of new neurons at each adaptive stage of the ANE method. This will be done by analyzing the physical meanings of neurons at each layer.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

神经网络在计算机视觉、自然语言处理和许多其他人工智能任务中取得了惊人的性能。尽管它们在许多实际应用中取得了巨大的成功，但人们普遍认为神经网络的近似特性尚未得到很好的理解。这个项目将有可能理解神经网络为什么以及如何工作，特别是为什么神经网络比其他函数类更好地用于计算机模拟奇异性和不连续性问题。在这个项目中开发的自适应神经网络方法将有可能显着提高计算机模拟复杂的物理，生物和人类工程系统表现出计算困难的计算能力。了解神经网络的宽度和深度在近似中的作用以及在该项目中获得的新想法将对许多其他人工智能任务产生重大影响，如迁移学习，在线控制和模式识别。预计至少有一名研究生将接受拟议研究课题的培训。机器学习中的一个基本的、开放的问题是如何设计神经网络的结构，根据它们的宽度和深度，以便准确有效地近似函数或数值求解偏微分方程。解决这个问题的应用程序计算上具有挑战性的问题是这个项目的重点。更确切地说，对于任何给定的偏微分方程，本项目将开发一种自适应神经元增强（ANE）方法，该方法自适应地构建具有几乎最小数量的神经元和参数的神经网络，使得其近似精度在规定的公差范围内。开发ANE方法的一个关键组成部分是用于确定要在当前层或下一层添加的新神经元数量的误差指示器。 该项目计划通过研究当前层和下一层中额外神经元的作用来开发有效的指标。这方面的知识是至关重要的设计有效和准确的ANE方法，但通常不提供标准的先验误差估计。神经网络卓越的逼近能力是有代价的：确定参数值的过程现在是非线性优化中的一个问题，即使潜在的偏微分方程是线性的。非线性优化通常有许多解，只有从足够接近的第一近似开始，才能得到所需的解。在本项目中开发的ANE方法是一个很好的延续过程。本计画将著重于如何在ANE方法的每个适应阶段初始化新神经元的参数。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Self-adaptive deep neural network: Numerical approximation to functions and PDEs

自适应深度神经网络：函数和偏微分方程的数值逼近

• DOI: 10.1016/j.jcp.2022.111021
• 发表时间: 2022
• 期刊: Journal of Computational Physics
• 影响因子: 4.1
• 作者: Cai, Zhiqiang;Chen, Jingshuang;Liu, Min
• 通讯作者: Liu, Min

Least-Squares ReLU Neural Network (LSNN) Method For Linear Advection-Reaction Equation

• DOI: 10.1016/j.jcp.2021.110514
• 发表时间: 2021-05
• 期刊: J. Comput. Phys.
• 作者: Z. Cai;Jingshuang Chen;Min Liu

Least-Squares ReLU Neural Network (LSNN) Method For Scalar Nonlinear Hyperbolic Conservation Law

• DOI: 10.1016/j.apnum.2022.01.002
• 发表时间: 2021-05
• 期刊: ArXiv
• 作者: Z. Cai;Jingshuang Chen;Min Liu

Least-squares neural network (LSNN) method for scalar nonlinear hyperbolic conservation laws: Discrete divergence operator

用于标量非线性双曲守恒定律的最小二乘神经网络 (LSNN) 方法：离散散度算子

• DOI: 10.1016/j.cam.2023.115298
• 发表时间: 2023
• 期刊: Journal of Computational and Applied Mathematics
• 影响因子: 2.4
• 作者: Cai, Zhiqiang;Chen, Jingshuang;Liu, Min
• 通讯作者: Liu, Min

Adaptive two-layer ReLU neural network: I. Best least-squares approximation

• DOI: 10.1016/j.camwa.2022.03.005
• 发表时间: 2022-03-17
• 期刊: COMPUTERS & MATHEMATICS WITH APPLICATIONS
• 影响因子: 2.9
• 作者: Liu, Min;Cai, Zhiqiang;Chen, Jingshuang
• 通讯作者: Chen, Jingshuang