Elastic Parameter Server: Accelerating ML Training With Scalable Resource Scheduling
Elastic Parameter Server: Accelerating ML Training With Scalable Resource Scheduling
复制标题
DOI：
10.1109/tpds.2021.3104242
复制
发表时间：
2022-05
影响因子：
5.3
通讯作者：
Shaoqi Wang;Aidi Pi;Xiaobo Zhou
中科院分区：
文献类型：
--
作者：
Shaoqi Wang;Aidi Pi;Xiaobo Zhou
针对集群环境下的分布式机器学习训练问题，设计了一种基于工作者-服务器通信的参数服务器。在ML模型训练的反馈驱动探索中，用户利用每个作业的早期反馈来决定是杀死作业还是保持其运行，以便找到最佳模型配置。但是，PS不支持在运行时调整作业的工作器和服务器的数量。它成为可扩展分布式ML训练的瓶颈，因为集群资源不能动态分配或释放到作业，导致显著的早期反馈延迟和资源利用不足。本文对PS体系结构的原理进行了反思。我们提出了弹性参数服务器（EPS），这是一个轻量级和用户透明的PS，可以加速分布式ML训练的反馈驱动探索。EPS允许从正在运行的作业中删除一部分工作者和服务器，并在运行时将释放的资源分配给传入的作业，以减少其早期反馈延迟。它还可以使用从被杀死的作业释放的资源来向正在运行的作业添加工人和服务器，以提高资源利用率和训练速度。我们开发了一个启发式调度器，它利用EPS，并为多个ML作业提供可扩展的资源调度。我们在Tencent Angel中实现EPS，在Apache Yarn中实现调度器，并使用各种ML模型进行评估。实验结果表明，与PS相比，EPS的ML训练速度提高了1.5倍。
Parameter server (PS) based on worker-server communication is designed for distributed machine learning (ML) training in clusters. In feedback-driven exploration of ML model training, users exploit early feedback from each job to decide whether to kill the job or keep it running so as to find the optimal model configuration. However, PS does not support adjusting the number of workers and servers of a job at runtime. It becomes the bottleneck of scalable distributed ML training because the cluster resources cannot be dynamically allocated or deallocated to jobs, resulting in significant early feedback latency and resource under-utilization. This article rethinks the principle of PS architecture. We present Elastic Parameter Server (EPS), a lightweight and user-transparent PS that accelerates feedback-driven exploration for distributed ML training. EPS allows to remove a subset of workers and servers from running jobs and allocate the released resources to an incoming job at runtime so as to reduce its early feedback latency. It can also use the released resources from a killed job to add workers and servers to running jobs to improve resource utilization and the training speed. We develop a heuristic scheduler that leverages EPS and offers scalable resource scheduling for multiple ML jobs. We implement EPS in Tencent Angel and the scheduler in Apache Yarn, and conduct evaluations with various ML models. Experimental results show that EPS achieves up to 1.5x improvement on the ML training speed compared to PS.