Predicting Throughput of Distributed Stochastic Gradient Descent
Predicting Throughput of Distributed Stochastic Gradient Descent
复制标题
DOI：
10.1109/tpds.2022.3151739
复制
发表时间：
2022
影响因子：
5.3
通讯作者：
Zhuojin Li;Marco Paolieri;L. Golubchik;Sung-Han Lin;Wumo Yan
中科院分区：
文献类型：
--
作者：
Zhuojin Li;Marco Paolieri;L. Golubchik;Sung-Han Lin;Wumo Yan
深度神经网络 (DNN) 的训练作业可以通过随机梯度下降 (SGD) 的分布式变体来加速，其中多个节点处理训练示例并交换更新。节点的总吞吐量不仅取决于它们的计算能力，还取决于它们的网络速度和协调机制（同步或异步、集中式或分散式），因为当配置更多节点时，通信瓶颈和落后者可能会导致次线性扩展。在本文中，我们提出了两类性能模型来预测分布式 SGD 的吞吐量：细粒度模型，表示许多基本计算/通信操作及其依赖关系；和粗粒度模型，其中每个节点的 SGD 步骤表示为一系列高级阶段，计算和通信之间没有并行性。使用 PyTorch 实现、真实世界的 DNN 模型和不同的云环境，我们的实验评估表明，虽然细粒度模型更准确并且可以轻松适应分布式 SGD 的新变体，但粗粒度模型在使用临时启发式增强时可以提供类似准确的预测，并且可以使用更容易收集的分析信息来估计其参数。
Training jobs of deep neural networks (DNNs) can be accelerated through distributed variants of stochastic gradient descent (SGD), where multiple nodes process training examples and exchange updates. The total throughput of the nodes depends not only on their computing power, but also on their networking speeds and coordination mechanism (synchronous or asynchronous, centralized or decentralized), since communication bottlenecks and stragglers can result in sublinear scaling when additional nodes are provisioned. In this paper, we propose two classes of performance models to predict throughput of distributed SGD: fine-grained models, representing many elementary computation/communication operations and their dependencies; and coarse-grained models, where SGD steps at each node are represented as a sequence of high-level phases without parallelism between computation and communication. Using a PyTorch implementation, real-world DNN models and different cloud environments, our experimental evaluation illustrates that, while fine-grained models are more accurate and can be easily adapted to new variants of distributed SGD, coarse-grained models can provide similarly accurate predictions when augmented with ad hoc heuristics, and their parameters can be estimated with profiling information that is easier to collect.