Accelerating TensorFlow with Adaptive RDMA-Based gRPC
Accelerating TensorFlow with Adaptive RDMA-Based gRPC
复制标题
DOI：
10.1109/hipc.2018.00010
复制
发表时间：
2018-12
期刊：
影响因子：
--
通讯作者：
Rajarshi Biswas;Xiaoyi Lu;D. Panda
中科院分区：
文献类型：
--
作者：
Rajarshi Biswas;Xiaoyi Lu;D. Panda
谷歌的TensorFlow是当今最受欢迎的深度学习框架之一。分布式TensorFlow支持多种通道来高效地传输张量，例如基于TCP/IP的GRPC、GRPC+动词和GRPC+MPI。目前，社区缺乏对分布式TensorFlow通信渠道的彻底刻画。这一点至关重要，因为TensorFlow的高性能深度学习需要高效的通信运行时。因此，我们对分布式TensorFlow的通信特性进行了深入的分析。我们的研究表明，TensorFlow中的现有通道都不能支持针对不同消息大小的深度学习工作负载的自适应高效通信。此外，社区需要维护这些不同的频道，同时用户也需要调整这些频道以获得期望的性能。因此，本文提出了一种统一的方法，利用自适应和高效的RDMA协议在TensorFlow中拥有单一的GRPC运行时(即AR-GRPC)。在AR-GRPC中，我们提出了混合通信协议、消息流水线和合并、零拷贝传输等设计，以使我们的运行时能够适应不同消息大小的深度学习负载。我们的性能评估表明，与IPoIB上的默认GRPC设计和社区中另一种基于RDMA的GRPC设计相比，AR-GRPC可以将GRPC性能显著提高4.1倍和2.3倍。Comet超级计算机显示，与默认的GRPC设计相比，AR-GRPC设计可以将点对点延迟减少高达75%。通过将我们的AR-GRPC与TensorFlow集成，我们可以实现比基于默认GRPC-IPoIB的TensorFlow高达3倍的分布式训练加速。
Google's TensorFlow is one of the most popular Deep Learning frameworks nowadays. Distributed TensorFlow supports various channels to efficiently transfer tensors, such as gRPC over TCP/IP, gRPC+Verbs, and gRPC+MPI. At present, the community lacks a thorough characterization of distributed TensorFlow communication channels. This is critical because high-performance Deep Learning with TensorFlow needs an efficient communication runtime. Thus, we conduct a thorough analysis of the communication characteristics of distributed TensorFlow. Our studies show that none of the existing channels in TensorFlow can support adaptive and efficient communication for Deep Learning workloads with different message sizes. Moreover, the community needs to maintain these different channels while the users are also expected to tune these channels to get the desired performance. Therefore, this paper proposes a unified approach to have a single gRPC runtime (i.e., AR-gRPC) in TensorFlow with Adaptive and efficient RDMA protocols. In AR-gRPC, we propose designs such as hybrid communication protocols, message pipelining and coalescing, zero-copy transmission etc. to make our runtime be adaptive to different message sizes for Deep Learning workloads. Our performance evaluations show that AR-gRPC can significantly speedup gRPC performance by up to 4.1x and 2.3x compared to the default gRPC design on IPoIB and another RDMA-based gRPC design in the community. Comet supercomputer shows that AR-gRPC design can reduce the Point-to-Point latency by up to 75% compared to the default gRPC design. By integrating our AR-gRPC with TensorFlow, we can achieve up to 3x distributed training speedup over default gRPC-IPoIB based TensorFlow.