Heavy Ball Neural Ordinary Differential Equations
Heavy Ball Neural Ordinary Differential Equations
复制标题
DOI：
--
复制
发表时间：
2021-10
期刊：
影响因子：
--
通讯作者：
Hedi Xia;Vai Suliafu;H. Ji;T. Nguyen;A. Bertozzi;S. Osher;Bao Wang
中科院分区：
文献类型：
--
作者：
Hedi Xia;Vai Suliafu;H. Ji;T. Nguyen;A. Bertozzi;S. Osher;Bao Wang
我们提出重球神经常微分方程（HBNODE），利用经典动量加速梯度下降的连续极限，来改进神经常微分方程（NODE）的训练和推理。 HBNODE 具有两个属性，这意味着比 NODE 具有实际优势：（i）HBNODE 的伴随状态也满足 HBNODE，加速前向和后向 ODE 求解器，从而显着减少函数评估（NFE）的数量并提高训练模型的实用性。 (ii) HBNODE 的范围结构良好，能够有效地从复杂的顺序数据中学习长期依赖性。我们在基准任务上验证了 HBNODE 相对于 NODE 的优势，包括图像分类、学习复杂动态和顺序建模。与其他基于 ODE 的神经网络模型相比，我们的方法需要的前向和后向 NFE 明显更少，更准确，并且更有效地学习长期依赖性。代码可在 \url{https://github.com/hedixia/HeavyBallNODE} 获取。
We propose heavy ball neural ordinary differential equations (HBNODEs), leveraging the continuous limit of the classical momentum accelerated gradient descent, to improve neural ODEs (NODEs) training and inference. HBNODEs have two properties that imply practical advantages over NODEs: (i) The adjoint state of an HBNODE also satisfies an HBNODE, accelerating both forward and backward ODE solvers, thus significantly reducing the number of function evaluations (NFEs) and improving the utility of the trained models. (ii) The spectrum of HBNODEs is well structured, enabling effective learning of long-term dependencies from complex sequential data. We verify the advantages of HBNODEs over NODEs on benchmark tasks, including image classification, learning complex dynamics, and sequential modeling. Our method requires remarkably fewer forward and backward NFEs, is more accurate, and learns long-term dependencies more effectively than the other ODE-based neural network models. Code is available at \url{https://github.com/hedixia/HeavyBallNODE}.