Derivation and analysis of parallel-in-time neural ordinary differential equations
Derivation and analysis of parallel-in-time neural ordinary differential equations
复制标题
并行时间神经常微分方程的推导与分析
DOI：
10.1007/s10472-020-09702-6
复制
发表时间：
2020
影响因子：
1.2
通讯作者：
E. Lorin
中科院分区：
文献类型：
--
作者：
E. Lorin
2015年引入的残差神经网络（RNN）和ResNET使得包含“大量”层的进化问题的学习算法的性能得到了显著改善。然后在2019年引入了称为神经常微分方程（NODE）的连续深度RNN模型。后者有一个恒定的内存成本，并避免了先验指定的隐藏层的数量。在本文中，我们推导和分析一个并行（在参数和时间）版本的节点，这可能会允许一个更有效的实现比标准/天真的并行化的节点只相对于参数。我们希望这种方法是相关的，每当我们有机会到一个非常大的数量的处理器，或者当我们正在处理高维ODE系统。此外，当使用隐式ODE求解器时，则需要使用例如牛顿算法来求解具有高达立方复杂度的线性系统的解;由于所提出的方法允许由于ODE系统求解器的精度阶的迭代增加而减少时间步长的总数，因此它减少了要求解的线性系统的数量，从而受益于缩放效应。
The introduction in 2015 of Residual Neural Networks (RNN) and ResNET allowed for outstanding improvements of the performance of learning algorithms for evolution problems containing a “large” number of layers. Continuous-depth RNN-like models called Neural Ordinary Differential Equations (NODE) were then introduced in 2019. The latter have a constant memory cost, and avoid the a priori specification of the number of hidden layers. In this paper, we derive and analyze a parallel (-in-parameter and time) version of the NODE, which potentially allows for a more efficient implementation than a standard/naive parallelization of NODEs with respect to the parameters only. We expect this approach to be relevant whenever we have access to a very large number of processors, or when we are dealing with high dimensional ODE systems. Moreover, when using implicit ODE solvers, solutions to linear systems with up to cubic complexity are then required for solving nonlinear systems using for instance Newton’s algorithm; as the proposed approach allows to reduce the overall number of time-steps thanks to an iterative increase of the accuracy order of the ODE system solvers, it then reduces the number of linear systems to solve, hence benefiting from a scaling effect.
可解释的多项式神经常微分方程。
DOI：
10.1063/5.0130803
发表时间：
2023
期刊：
Chaos (Woodbury, N.Y.)
影响因子：
--
作者：
Fronk,Colby;Petzold,Linda
通讯作者：
Petzold,Linda