Deep Neural Network Training with Distributed K-FAC
Deep Neural Network Training with Distributed K-FAC
复制标题
DOI：
10.1109/tpds.2022.3161187
复制
发表时间：
2022
影响因子：
5.3
通讯作者：
J. G. Pauloski;Lei Huang;Weijia Xu;K. Chard;I. Foster;Zhao Zhang
中科院分区：
文献类型：
--
作者：
J. G. Pauloski;Lei Huang;Weijia Xu;K. Chard;I. Foster;Zhao Zhang
将深层神经网络培训扩展到更多的处理器和更大的批量大小是减少端到端训练时间的关键；但是，在较大尺度上保持可比的收敛和硬件利用是一个挑战。训练量表的增加使自然梯度优化方法是SGD及其变体的合理替代方法。 KRONECKER-FACTER近似曲率（K-FAC）是一种天然梯度方法，其前提是具有有效近似Fisher Information矩阵的梯度，以在优化目标函数时改善每卷曲的进展。在这项工作中，我们提出了一种可扩展的K-FAC算法，并研究了大型深神经网络培训中的K-FACS适用性。具体来说，我们探索了层的分配策略，无反向的二阶梯度评估以及动态K-FAC更新解耦，以保持收敛的目的，同时最大程度地减少训练时间。使用图像分类，对象检测和语言建模域中的应用进行了有关我们K-FAC梯度预处理器的收敛性和缩放特性的研究。在所有应用中，我们的实现都比在多种尺度上的GPU簇上的标准一阶优化器降低了925％的基线性能目标
Scaling deep neural network training to more processors and larger batch sizes is key to reducing end-to-end training time; yet, maintaining comparable convergence and hardware utilization at larger scales is a challenge. Increases in training scales has enabled natural gradient optimization methods as a reasonable alternative to SGD and variants thereof. Kronecker-factored Approximate Curvature (K-FAC), a natural gradient method, preconditions gradients with an efficient approximation of the Fisher Information Matrix to improve per-iteration progress when optimizing an objective function. In this work, we propose a scalable K-FAC algorithm and investigate K-FACs applicability in large-scale deep neural network training. Specifically, we explore layer-wise distribution strategies, inverse-free second-order gradient evaluation, and dynamic K-FAC update decoupling with the goal of preserving convergence while minimizing training time. A study on the convergence and scaling properties of our K-FAC gradient preconditioner is presented using applications in the image classification, object detection, and language modeling domains. In all applications, our implementation converges to baseline performance targets in 925% less than the standard first-order optimizers on GPU clusters across a variety scales