HoD-Net: High-Order Differentiable Deep Neural Networks and Applications
HoD-Net: High-Order Differentiable Deep Neural Networks and Applications
复制标题
DOI：
10.1609/aaai.v36i8.20799
复制
发表时间：
2022-06
期刊：
影响因子：
--
通讯作者：
Siyuan Shen;Tianjia Shao;Kun Zhou;Chenfanfu Jiang;Feng Luo;Yin Yang
中科院分区：
文献类型：
--
作者：
Siyuan Shen;Tianjia Shao;Kun Zhou;Chenfanfu Jiang;Feng Luo;Yin Yang
我们引入了一个名为HoD-Net的深度架构，以实现深度学习的高阶可微性。HoD-Net基于并推广了复步长有限差分（CSFD）方法。虽然与经典的有限差分类似，但CSFD从更高维的复域中逼近函数的导数，从而实现高度准确和鲁棒的微分计算，而不会出现数值稳定性问题。该方法可以与反向传播和伴随摄动法相结合，有效地计算高阶导数。我们展示了如何利用这种数值方案来解决具有挑战性的深度学习问题，例如高阶网络训练、基于深度学习的物理模拟和神经微分方程。
We introduce a deep architecture named HoD-Net to enable high-order differentiability for deep learning. HoD-Net is based on and generalizes the complex-step finite difference (CSFD) method. While similar to classic finite difference, CSFD approaches the derivative of a function from a higher-dimension complex domain, leading to highly accurate and robust differentiation computation without numerical stability issues. This method can be coupled with backpropagation and adjoint perturbation methods for an efficient calculation of high-order derivatives. We show how this numerical scheme can be leveraged in challenging deep learning problems, such as high-order network training, deep learning-based physics simulation, and neural differential equations.