Scalable and Practical Natural Gradient for Large-Scale Deep Learning
Scalable and Practical Natural Gradient for Large-Scale Deep Learning
复制标题
DOI：
10.1109/tpami.2020.3004354
复制
发表时间：
2022-01-01
影响因子：
23.6
通讯作者：
Yokota, Rio
中科院分区：
文献类型：
--
作者：
Osawa, Kazuki;Tsuji, Yohei;Yokota, Rio
深度神经网络的大规模分布式训练由于有效小批大小的增加而导致模型泛化性能变差。以前的方法试图通过改变学习率和批大小来解决这个问题，或者对批规范化进行特别修改。我们提出了可扩展和实用的自然梯度下降（SP-NGD），这是一种训练模型的原则方法，使它们能够获得与使用一阶优化方法训练的模型相似的泛化性能，但具有加速收敛性。此外，与一阶方法相比，SP-NGD可以扩展到大的小批量大小，计算开销可以忽略不计。我们在一个基准任务上评估了SP-NGD，其中高度优化的一阶方法可用作参考：在ImageNet上训练ResNet-50模型用于图像分类。我们展示了在5.5分钟内收敛到75.4%的顶级验证精度，使用32,768个小批量和1,024个gpu，以及在SP-NGD的873步中使用131,072个超大小批量的74.9%的精度。
Large-scale distributed training of deep neural networks results in models with worse generalization performance as a result of the increase in the effective mini-batch size. Previous approaches attempt to address this problem by varying the learning rate and batch size over epochs and layers, or ad hoc modifications of batch normalization. We propose scalable and practical natural gradient descent (SP-NGD), a principled approach for training models that allows them to attain similar generalization performance to models trained with first-order optimization methods, but with accelerated convergence. Furthermore, SP-NGD scales to large mini-batch sizes with a negligible computational overhead as compared to first-order methods. We evaluated SP-NGD on a benchmark task where highly optimized first-order methods are available as references: training a ResNet-50 model for image classification on ImageNet. We demonstrate convergence to a top-1 validation accuracy of 75.4 percent in 5.5 minutes using a mini-batch size of 32,768 with 1,024 GPUs, as well as an accuracy of 74.9 percent with an extremely large mini-batch size of 131,072 in 873 steps of SP-NGD.