SimiGrad: Fine-Grained Adaptive Batching for Large Scale Training using Gradient Similarity Measurement
SimiGrad: Fine-Grained Adaptive Batching for Large Scale Training using Gradient Similarity Measurement
复制标题
DOI：
--
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Heyang Qin;Samyam Rajbhandari;Olatunji Ruwase;Feng Yan;Lei Yang;Yuxiong He
中科院分区：
文献类型：
--
作者：
Heyang Qin;Samyam Rajbhandari;Olatunji Ruwase;Feng Yan;Lei Yang;Yuxiong He
大规模训练需要大量并行性才能在合理的时间内完成训练。为了支持大规模并行性，大批量训练是关键的推动因素，但通常以牺牲泛化性能为代价。现有的工作探索自适应批处理或手动调整的静态大批处理，以在计算效率和性能之间取得平衡。然而，由于固有的昂贵的计算或手动调整要求，这些方法只能提供粗粒度的适应（例如，在纪元级别）。在本文中，我们提出了一种完全自动化且轻量级的自适应批处理方法，以实现细粒度的批量大小调整（例如，在小批量级别），从而可以以破纪录的批量大小实现最先进的性能。我们方法的核心组成部分是关键梯度噪声信息的轻量级但有效的表示。我们通过提供支持主流机器学习框架的插件工具来开源所提出的方法。对流行基准（例如 CIFAR10、ImageNet 和 BERT-Large）的广泛评估表明，所提出的方法在性能和批量大小方面均优于使用自适应批处理方法或手动调整静态策略的最先进方法。特别是，我们在 BERT-Large 预训练中实现了新的最先进的批量大小 78k，SQuAD 得分为 90.69，而之前最先进的 59k 批量大小的得分为 90.58。
Large scale training requires massive parallelism to finish the training within a reasonable amount of time. To support massive parallelism, large batch training is the key enabler but often at the cost of generalization performance. Existing works explore adaptive batching or hand-tuned static large batching, in order to strike a balance between the computational efficiency and the performance. However, these methods can provide only coarse-grained adaption (e.g., at a epoch level) due to the intrinsic expensive calculation or hand tuning requirements. In this paper, we propose a fully automated and lightweight adaptive batching methodology to enable fine-grained batch size adaption (e.g., at a mini-batch level) that can achieve state-of-the-art performance with record breaking batch sizes. The core component of our method is a lightweight yet efficient representation of the critical gradient noise information. We open-source the proposed methodology by providing a plugin tool that supports mainstream machine learning frameworks. Extensive evaluations on popular benchmarks (e.g., CIFAR10, ImageNet, and BERT-Large) demonstrate that the proposed methodology outperforms state-of-the-art methodologies using adaptive batching approaches or hand-tuned static strategies in both performance and batch size. Particularly, we achieve a new state-of-the-art batch size of 78k in BERT-Large pretraining with SQuAD score 90.69 compared to 90.58 reported in previous state-of-the-art with 59k batch size.