On scheduling ring-all-reduce learning jobs in multi-tenant GPU clusters with communication contention
On scheduling ring-all-reduce learning jobs in multi-tenant GPU clusters with communication contention
复制标题
DOI：
10.1145/3492866.3549716
复制
发表时间：
2022-07
期刊：
影响因子：
--
通讯作者：
Menglu Yu;Bo Ji;Hridesh Rajan;Jia Liu
中科院分区：
文献类型：
--
作者：
Menglu Yu;Bo Ji;Hridesh Rajan;Jia Liu
得益于深度学习（DL）技术的进步，机器学习和人工智能取得了惊人的成功。然而，对深度学习日益增长的需求也导致了大规模深度学习训练中通信和资源密集型的分布式训练任务，这些任务通常部署在GPU集群上。为了满足对深度学习训练不断增长的需求，所谓的“环全归约”（RAR）技术最近作为一种有利的计算架构应运而生，它能有效地处理GPU集群中的网络通信和计算负载。RAR最显著的特征是它不需要专用的参数服务器，从而缓解了潜在的通信瓶颈。然而，当多个基于RAR的深度学习训练任务部署在GPU集群上时，由于深度学习训练任务之间的竞争，通信瓶颈仍然可能出现。到目前为止，对于如何为基于RAR的深度学习训练任务设计具有竞争感知的资源调度算法，仍然缺乏理论上的理解，这促使我们在这项工作中填补这一空白。我们的主要贡献有三点：i）我们开发了一种新的分析模型，该模型既能描述与任务的工作节点分布相关的通信开销，又能描述与不同任务共置相关的通信竞争；ii）基于所提出的分析模型，我们将问题表述为一个非凸整数规划，以最小化所有基于RAR的深度学习训练任务的完工时间。为了解决这个问题中不利于优化算法设计的独特结构，我们将问题重新表述为一个整数线性规划，从而能够设计出一种可证明的近似算法，称为SJF - BCO（具有平衡竞争和开销的最短作业优先）；iii）我们进行了大量实验，以证明SJF - BCO相对于现有调度器的优越性。总体而言，我们的研究结果有助于分布式GPU系统优化和算法设计的前沿发展。
Powered by advances in deep learning (DL) techniques, machine learning and artificial intelligence have achieved astonishing successes. However, the rapidly growing needs for DL also led to communication- and resource-intensive distributed training jobs for large-scale DL training, which are typically deployed over GPU clusters. To sustain the ever-increasing demand for DL training, the so-called "ring-all-reduce" (RAR) technologies have recently emerged as a favorable computing architecture to efficiently process network communication and computation load in GPU clusters. The most salient feature of RAR is that it removes the need for dedicated parameter servers, thus alleviating the potential communication bottleneck. However, when multiple RAR-based DL training jobs are deployed over GPU clusters, communication bottlenecks could still occur due to contentions between DL training jobs. So far, there remains a lack of theoretical understanding on how to design contention-aware resource scheduling algorithms for RAR-based DL training jobs, which motivates us to fill this gap in this work. Our main contributions are three-fold: i) We develop a new analytical model that characterizes both communication overhead related to the worker distribution of the job and communication contention related to the co-location of different jobs; ii) Based on the proposed analytical model, we formulate the problem as a non-convex integer program to minimize the makespan of all RAR-based DL training jobs. To address the unique structure in this problem that is not amenable for optimization algorithm design, we reformulate the problem into an integer linear program that enables provable approximation algorithm design called SJF-BCO (Smallest Job First with Balanced Contention and Overhead); and iii) We conduct extensive experiments to show the superiority of SJF-BCO over existing schedulers. Collectively, our results contribute to the state-of-the-art of distributed GPU system optimization and algorithm design.