Distributed Proximal Gradient Algorithm for Partially Asynchronous Computer Clusters
Distributed Proximal Gradient Algorithm for Partially Asynchronous Computer Clusters
复制标题
DOI：
--
复制
发表时间：
2017-04
期刊：
影响因子：
--
通讯作者：
Yi Zhou;Yingbin Liang;Yaoliang Yu;Wei Dai;E. Xing
中科院分区：
文献类型：
--
作者：
Yi Zhou;Yingbin Liang;Yaoliang Yu;Wei Dai;E. Xing
随着数据量和模型规模的不断增长，容错、通信效率高、通用性强的分布式算法对于许多大规模机器学习应用的成功至关重要。在这项工作中，我们提出了m-PAPG，灵活的近端梯度算法在模型并行系统配备了部分异步通信协议的实现。工作机器与受控的陈旧界限异步通信，并以不同的频率运行。我们刻画了m-PAPG的各种收敛性质：1）在一般的非光滑非凸条件下，证明了m-PAPG产生的序列的每个极限点都是目标函数的临界点; 2）在凸目标函数的误差有界条件下，证明了最优性间隙每s步线性衰减; 3）在Kurdyka-Kazojasiewicz不等式和充分减少的假设下，证明了m-PAPG生成的序列在满足邻近Lipschitz条件下收敛于同一临界点。
With ever growing data volume and model size, an error-tolerant, communication efficient, yet versatile distributed algorithm has become vital for the success of many large-scale machine learning applications. In this work we propose m-PAPG, an implementation of the flexible proximal gradient algorithm in model parallel systems equipped with the partially asynchronous communication protocol. The worker machines communicate asynchronously with a controlled staleness bound s and operate at different frequencies. We characterize various convergence properties of m-PAPG: 1) Under a general non-smooth and nonconvex setting, we prove that every limit point of the sequence generated by m-PAPG is a critical point of the objective function; 2) Under an error bound condition of convex objective functions, we prove that the optimality gap decays linearly for every s steps; 3) Under the Kurdyka-Łojasiewicz inequality and a sufficient decrease assumption, we prove that the sequences generated by m-PAPG converge to the same critical point, provided that a proximal Lipschitz condition is satisfied.