Layer-Parallel Training of Deep Residual Neural Networks
Layer-Parallel Training of Deep Residual Neural Networks
复制标题
DOI：
10.1137/19m1247620
复制
发表时间：
2018-12
期刊：
影响因子：
--
通讯作者：
Stefanie Günther;Lars Ruthotto;J. Schroder;E. Cyr;N. Gauger
中科院分区：
文献类型：
--
作者：
Stefanie Günther;Lars Ruthotto;J. Schroder;E. Cyr;N. Gauger
残差神经网络（ResNets）是一类很有前景的深度神经网络，在诸多学习任务（如图像分类与识别）中表现卓越。从数学角度看，ResNet架构可被理解为一个非线性初值问题的前向欧拉离散化，其中与时间相关的控制变量代表神经网络的权重。因此，训练ResNet可被转化为相关动力系统的最优控制问题。对于工程应用中出现的类似与时间相关的最优控制问题，时间并行方法在可扩展性方面已展现出显著提升。本文展示了如何运用这些技术对ResNets进行高效且有效的训练。所提出的算法通过将应用于层域的并行非线性多重网格迭代，取代经典的（顺序的）逐层前向和反向传播。这在层间增加了一个新的并行维度，在训练极深网络时颇具吸引力。基于这一基本思路，我们推导出多种层并行方法。其中最有效的版本采用同步优化方法，即对网络参数的更新基于不精确的梯度信息，以加快训练过程。通过监督分类的数值示例，我们证明新方法能实现与传统方法类似的训练性能，但支持层并行，因而通过更高的并发性，相比层串行方法实现加速。
Residual neural networks (ResNets) are a promising class of deep neural networks that have shown excellent performance for a number of learning tasks, e.g., image classification and recognition. Mathematically, ResNet architectures can be interpreted as forward Euler discretizations of a nonlinear initial value problem whose time-dependent control variables represent the weights of the neural network. Hence, training a ResNet can be cast as an optimal control problem of the associated dynamical system. For similar time-dependent optimal control problems arising in engineering applications, parallel-in-time methods have shown notable improvements in scalability. This paper demonstrates the use of those techniques for efficient and effective training of ResNets. The proposed algorithms replace the classical (sequential) forward and backward propagation through the network layers by a parallel nonlinear multigrid iteration applied to the layer domain. This adds a new dimension of parallelism across layers that is attractive when training very deep networks. From this basic idea, we derive multiple layer-parallel methods. The most efficient version employs a simultaneous optimization approach where updates to the network parameters are based on inexact gradient information in order to speed up the training process. Using numerical examples from supervised classification, we demonstrate that the new approach achieves similar training performance to traditional methods, but enables layer-parallelism and thus provides speedup over layer-serial methods through greater concurrency.