A distributed stochastic optimization algorithm with gradient-tracking and distributed heavy-ball acceleration
A distributed stochastic optimization algorithm with gradient-tracking and distributed heavy-ball acceleration
复制标题
具有梯度跟踪和分布式重球加速的分布式随机优化算法
DOI：
10.1631/fitee.2000615
复制
发表时间：
2021-07
影响因子：
3
通讯作者：
Bihao Sun;Jinhui Hu;Dawen Xia;Huaqing Li
中科院分区：
文献类型：
--
作者：
Bihao Sun;Jinhui Hu;Dawen Xia;Huaqing Li
分布式优化由于其在机器学习和信号处理中的广泛应用，近年来得到了很好的发展。在本文中，我们主要研究最小化全局目标的分布式优化问题。目标是分布在n个节点的无向网络上的光滑和强凸的局部代价函数的和。与已有工作相比，我们采用了分布式重球项来提高算法的收敛性能。为了加快现有分布式随机一阶梯度法的收敛速度，将动量项与梯度跟踪技术相结合。实验结果表明，该算法在不增加算法复杂度的前提下，具有比GT-SAGA更好的加速能力。在真实数据集上的大量实验验证了该算法的有效性和正确性。
Distributed optimization has been well developed in recent years due to its wide applications in machine learning and signal processing. In this paper, we focus on investigating distributed optimization to minimize a global objective. The objective is a sum of smooth and strongly convex local cost functions which are distributed over an undirected network of n nodes. In contrast to existing works, we apply a distributed heavy-ball term to improve the convergence performance of the proposed algorithm. To accelerate the convergence of existing distributed stochastic first-order gradient methods, a momentum term is combined with a gradient-tracking technique. It is shown that the proposed algorithm has better acceleration ability than GT-SAGA without increasing the complexity. Extensive experiments on real-world datasets verify the effectiveness and correctness of the proposed algorithm.