A Proactive Data-Parallel Framework for Machine Learning
A Proactive Data-Parallel Framework for Machine Learning
复制标题
DOI：
10.1145/3492324.3494167
复制
发表时间：
2021-12
期刊：
影响因子：
--
通讯作者：
Guoyi Zhao;Tian Zhou;Lixin Gao
中科院分区：
文献类型：
--
作者：
Guoyi Zhao;Tian Zhou;Lixin Gao
数据并行框架对于训练机器学习模型至关重要。经典的批量同步并行（BSP）模型通过预定义的同步屏障更新模型参数。但是，当一个worker的计算速度明显慢于其他worker时，等待这个慢的worker将导致计算资源的过度浪费。在本文中，我们提出了一个新的主动数据并行（PDP）框架。PDP使参数服务器能够初始化模型参数的更新。也就是说，我们可以在没有预定义更新点的情况下随时执行更新。PDP不仅发起更新，而且还决定何时更新。关于更新频率的全球决定将加速培训。我们进一步提出异步PDP，以减少同步参数更新造成的空闲时间。从理论上证明了异步PDP算法的收敛性。我们实现了一个分布式PDP框架，并使用几种流行的机器学习算法（包括多层感知器、卷积神经网络、K-means和高斯混合模型）来评估PDP。我们的评估表明，与BSP模型相比，PDP可以实现高达20倍的加速，并且可以扩展到大型集群。
Data parallel frameworks become essential for training machine learning models. The classic Bulk Synchronous Parallel (BSP) model updates the model parameters through pre-defined synchronization barriers. However, when a worker computes significantly slower than other workers, waiting for the slow worker will lead to excessive waste of computing resources. In this paper, we propose a novel proactive data-parallel (PDP) framework. PDP enables the parameter server to initiate the update of the model parameter. That is, we can perform the update at any time without pre-defined update points. PDP not only initiates the update but also determines when to update. The global decision on the frequency of updates will accelerate the training. We further propose asynchronous PDP to reduce the idle time caused by synchronizing parameter updates. We theoretically prove the convergence property of asynchronous PDP. We implement a distributed PDP framework and evaluate PDP with several popular machine learning algorithms including Multilayer Perceptron, Convolutional Neural Network, K-means, and Gaussian Mixture Model. Our evaluation shows that PDP can achieve up to 20X speedup over the BSP model and scale to large clusters.