TicTac: Accelerating Distributed Deep Learning with Communication Scheduling
TicTac: Accelerating Distributed Deep Learning with Communication Scheduling
复制标题
DOI：
--
复制
发表时间：
2018-03
期刊：
影响因子：
--
通讯作者：
Sayed Hadi Hashemi;S. Jyothi;R. Campbell
中科院分区：
文献类型：
--
作者：
Sayed Hadi Hashemi;S. Jyothi;R. Campbell
最先进的深度学习系统依赖于迭代分布式训练来解决模型和输入数据日益复杂的问题。在这些通信繁重的系统中，迭代时间取决于计算时间、通信时间以及计算和通信的重叠程度。在这项工作中，我们发现了使用图形表示进行计算的系统（如TensorFlow和PyTorch）中的一个缺点，这导致迭代时间的高方差-工人之间接收参数的随机顺序。我们开发了一个系统TicTac，通过使用参数服务器解决分布式深度学习中的这个问题来改善迭代时间，同时保证通信和计算的接近最佳重叠。TicTac识别并强制执行网络传输的顺序，从而使用优先级提高迭代时间。我们的系统是通过TensorFlow实现的，不需要对模型或开发人员输入进行更改。TicTac在推理方面提高了高达37.7\%$的吞吐量，在训练方面提高了19.2\%$，同时还减少了高达2.3\times $的落伍者效应。我们的代码是公开的。
State-of-the-art deep learning systems rely on iterative distributed training to tackle the increasing complexity of models and input data. The iteration time in these communication-heavy systems depends on the computation time, communication time and the extent of overlap of computation and communication. In this work, we identify a shortcoming in systems with graph representation for computation, such as TensorFlow and PyTorch, that result in high variance in iteration time --- random order of received parameters across workers. We develop a system, TicTac, to improve the iteration time by fixing this issue in distributed deep learning with Parameter Servers while guaranteeing near-optimal overlap of communication and computation. TicTac identifies and enforces an order of network transfers which improves the iteration time using prioritization. Our system is implemented over TensorFlow and requires no changes to the model or developer inputs. TicTac improves the throughput by up to $37.7\%$ in inference and $19.2\%$ in training, while also reducing straggler effect by up to $2.3\times$. Our code is publicly available.