Randomized Sparse Neural Galerkin Schemes for Solving Evolution Equations with Deep Networks
Randomized Sparse Neural Galerkin Schemes for Solving Evolution Equations with Deep Networks
复制标题
DOI：
10.48550/arxiv.2310.04867
复制
发表时间：
2023-10
期刊：
影响因子：
--
通讯作者：
Jules Berman;Benjamin Peherstorfer
中科院分区：
文献类型：
--
作者：
Jules Berman;Benjamin Peherstorfer
按时间顺序训练神经网络以近似时间相关偏微分方程的解场可以有利于保持因果关系和其他物理属性;然而，按时间顺序训练在数值上具有挑战性，因为训练误差会随着时间的推移迅速积累和放大。这项工作介绍了神经Galerkin计划，更新随机稀疏的网络参数的子集在每个时间步。随机化避免了时间上的局部过拟合，因此有助于防止误差在时间上的顺序训练中快速积累，这是由dropout引起的，dropout解决了由于神经元协适应引起的过拟合的类似问题。更新的稀疏性降低了训练的计算成本，而不会失去表现力，因为许多网络参数在每个时间步都是局部冗余的。在广泛的演化方程的数值实验中，所提出的计划与随机稀疏更新是高达两个数量级更准确，在一个固定的计算预算和高达两个数量级更快，在一个固定的精度比密集的更新计划。
Training neural networks sequentially in time to approximate solution fields of time-dependent partial differential equations can be beneficial for preserving causality and other physics properties; however, the sequential-in-time training is numerically challenging because training errors quickly accumulate and amplify over time. This work introduces Neural Galerkin schemes that update randomized sparse subsets of network parameters at each time step. The randomization avoids overfitting locally in time and so helps prevent the error from accumulating quickly over the sequential-in-time training, which is motivated by dropout that addresses a similar issue of overfitting due to neuron co-adaptation. The sparsity of the update reduces the computational costs of training without losing expressiveness because many of the network parameters are redundant locally at each time step. In numerical experiments with a wide range of evolution equations, the proposed scheme with randomized sparse updates is up to two orders of magnitude more accurate at a fixed computational budget and up to two orders of magnitude faster at a fixed accuracy than schemes with dense updates.