On robustness of neural ODEs image classifiers
On robustness of neural ODEs image classifiers
复制标题
DOI：
10.1016/j.ins.2023.03.049
复制
发表时间：
2023-03
期刊：
影响因子：
--
通讯作者：
Wenjun Cui;Honglei Zhang;Haoyu Chu;Pipi Hu;Yidong Li
中科院分区：
文献类型：
--
作者：
Wenjun Cui;Honglei Zhang;Haoyu Chu;Pipi Hu;Yidong Li
神经常微分方程（Neural奥德斯）作为一类新颖的深层模型，将传统神经网络与动力系统巧妙地联系起来，在理论与实践之间架起了一座桥梁。然而，他们还没有在激活函数上取得实质性的进展，并且默认情况下总是使用ReLU。而且，随着训练的进行，其中存在的动态行为也变得越来越模糊和复杂。幸运的是，现有的研究表明，激活功能是神经ode控制内在动力学的必要条件。基于一组增强动力系统稳定性的权函数，我们引入了一种新的激活函数half-Swish来匹配神经ode。此外，我们还探讨了演化时间和批处理大小对神经ode的影响。实验表明，在Fashion-MNIST、CIFAR-10和CIFAR-100数据集上，我们的模型对随机噪声图像和对抗样本的鲁棒性都优于具有基本激活函数的Neural des，这有力地验证了half-Swish的适用性，并表明half-Swish函数在规范动态行为以增强稳定性方面发挥了积极作用。同时，我们的工作在理论上为选择合适的激活函数来匹配神经微分方程提供了一个有前景的框架。
Neural Ordinary Differential Equations (Neural ODEs), as a family of novel deep models, delicately link conventional neural networks and dynamical systems, which bridges the gap between theory and practice. However, they have not made substantial progress on activation functions, and ReLU is always utilized by default. Moreover, the dynamical behavior existing in them becomes more unclear and complicated as training progresses. Fortunately, existing studies have shown that activation functions are essential for Neural ODEs in governing intrinsic dynamics. Motivated by a family of weight functions used to enhance the stability of dynamical systems, we introduce a new activation function named half-Swish to match Neural ODEs. Besides, we explore the effect of evolution time and batch size on Neural ODEs, respectively. Experiments show that our model consistently outperforms Neural ODEs with basic activation functions on robustness both against stochastic noise images and adversarial examples across Fashion-MNIST, CIFAR-10, and CIFAR-100 datasets, which strongly validates the applicability of half-Swish and suggests that half-Swish function plays a positive role in regularizing the dynamic behavior to enhance stability. Meanwhile, our work theoretically provides a prospective framework to choose appropriate activation functions to match neural differential equations.