Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism
Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism
复制标题
DOI：
10.1016/j.jcp.2022.111722
复制
发表时间：
2020-09
期刊：
影响因子：
--
通讯作者：
L. McClenny;U. Braga-Neto
中科院分区：
文献类型：
--
作者：
L. McClenny;U. Braga-Neto
物理信息神经网络 (PINN) 最近出现，作为深度神经网络在非线性偏微分方程 (PDE) 数值求解中的一种有前景的应用。然而，人们已经认识到需要自适应程序来迫使神经网络准确地适应“刚性”偏微分方程解中的顽固点。在本文中，我们提出了一种自适应训练 PINN 的全新方法，其中自适应权重是完全可训练的，并单独应用于每个训练点，因此神经网络可以自主学习解决方案中的哪些区域是困难的，并被迫关注它们。自适应权重指定了软乘法软注意掩模，这让人想起计算机视觉中使用的类似机制。这些 SA-PINN 背后的基本思想是使权重随着相应损失的增加而增加，这是通过训练网络同时最小化损失和最大化权重来实现的。此外，我们还展示了如何使用高斯过程回归构建自适应权重的连续图，这允许在传统梯度下降不足以产生准确解决方案的问题中使用随机梯度下降。最后，我们推导了 SA-PINN 的神经正切核矩阵，并使用它来获得对无限宽 PINN 极限情况下自适应权重对训练动态的影响的启发式理解，这表明 SA-PINN 的工作原理是产生与不同损失项相对应的 NTK 矩阵特征值的平滑均衡。在几个线性和非线性基准问题的数值实验中，SA-PINN 在 L2 误差方面优于其他最先进的 PINN 算法，同时使用较少数量的训练周期。
Physics-Informed Neural Networks (PINNs) have emerged recently as a promising application of deep neural networks to the numerical solution of nonlinear partial differential equations (PDEs). However, it has been recognized that adaptive procedures are needed to force the neural network to fit accurately the stubborn spots in the solution of “stiff” PDEs. In this paper, we propose a fundamentally new way to train PINNs adaptively, where the adaptation weights are fully trainable and applied to each training point individually, so the neural network learns autonomously which regions of the solution are difficult and is forced to focus on them. The self-adaptation weights specify a soft multiplicative soft attention mask, which is reminiscent of similar mechanisms used in computer vision. The basic idea behind these SA-PINNs is to make the weights increase as the corresponding losses increase, which is accomplished by training the network to simultaneously minimize the losses and maximize the weights. In addition, we show how to build a continuous map of self-adaptive weights using Gaussian Process regression, which allows the use of stochastic gradient descent in problems where conventional gradient descent is not enough to produce accurate solutions. Finally, we derive the Neural Tangent Kernel matrix for SA-PINNs and use it to obtain a heuristic understanding of the effect of the self-adaptive weights on the dynamics of training in the limiting case of infinitely-wide PINNs, which suggests that SA-PINNs work by producing a smooth equalization of the eigenvalues of the NTK matrix corresponding to the different loss terms. In numerical experiments with several linear and nonlinear benchmark problems, the SA-PINN outperformed other state-of-the-art PINN algorithm in L2 error, while using a smaller number of training epochs.