PyTorch Distributed: Experiences on Accelerating Data Parallel Training
PyTorch Distributed: Experiences on Accelerating Data Parallel Training
复制标题
DOI：
10.14778/3415478.3415530
复制
发表时间：
2020-08-01
影响因子：
2.5
通讯作者：
Chintala, Soumith
中科院分区：
文献类型：
--
作者：
Li, Shen;Zhao, Yanli;Chintala, Soumith
本文介绍了PyTorch分布式数据并行模块的设计、实现和评估。PyTorch是一个广泛采用的科学计算包，用于深度学习研究和应用。深度学习的最新进展证明了大型数据集和大型模型的价值，这需要将模型训练扩展到更多计算资源的能力。由于其简单的原理和广泛的适用性，数据并行已经成为分布式训练的流行解决方案。一般来说，分布式数据并行技术在每个计算资源上复制模型以独立地生成梯度，然后在每次迭代时传递这些梯度以保持模型副本一致。尽管该技术在概念上很简单，但计算和通信之间的微妙依赖关系使得优化分布式训练效率变得非常重要。从v1.5开始，PyTorch原生地提供了几种技术来加速分布式数据并行，包括分级梯度，重叠计算与通信，以及跳过梯度同步。评估表明，在适当配置时，PyTorch分布式数据并行模块使用256个GPU实现了接近线性的可扩展性。
This paper presents the design, implementation, and evaluation of the PyTorch distributed data parallel module. PyTorch is a widely-adopted scientific computing package used in deep learning research and applications. Recent advances in deep learning argue for the value of large datasets and large models, which necessitates the ability to scale out model training to more computational resources. Data parallelism has emerged as a popular solution for distributed training thanks to its straightforward principle and broad applicability. In general, the technique of distributed data parallelism replicates the model on every computational resource to generate gradients independently and then communicates those gradients at each iteration to keep model replicas consistent. Despite the conceptual simplicity of the technique, the subtle dependencies between computation and communication make it non-trivial to optimize the distributed training efficiency. As of v1.5, PyTorch natively provides several techniques to accelerate distributed data parallel, including bucketing gradients, overlapping computation with communication, and skipping gradient synchronization. Evaluations show that, when configured appropriately, the PyTorch distributed data parallel module attains near-linear scalability using 256 GPUs.