SignProx: One-bit Proximal Algorithm for Nonconvex Stochastic Optimization
SignProx: One-bit Proximal Algorithm for Nonconvex Stochastic Optimization
复制标题
DOI：
10.1109/icassp.2019.8682059
复制
发表时间：
2018-07
期刊：
影响因子：
--
通讯作者：
U. Kamilov
中科院分区：
文献类型：
--
作者：
U. Kamilov
随机梯度下降算法（SGD）是应用最广泛的大型数据集并行和分布式处理优化方法之一。分布式SGD的主要限制之一是需要在不同计算节点之间定期通信梯度。为了减少这种通信瓶颈，最近的工作考虑了SGD的一位变体，其中仅使用每个梯度元素的符号进行优化。在本文中，我们扩展了这个想法，提出了一种随机变体的近端梯度方法，每个更新元素也使用一个比特。在一组显式假设下，证明了该方法的理论收敛性。我们的结果表明，压缩方法的收敛速度可以达到未压缩方法的收敛速度，这使得所提出的方法对大型数据集的分布式处理具有潜在的吸引力。
Stochastic gradient descent (SGD) is one of the most widely used optimization methods for parallel and distributed processing of large datasets. One of the key limitations of distributed SGD is the need to regularly communicate the gradients between different computation nodes. To reduce this communication bottleneck, recent work has considered a one-bit variant of SGD, where only the sign of each gradient element is used in optimization. In this paper, we extend this idea by proposing a stochastic variant of the proximal-gradient method that also uses one-bit per update element. We prove the theoretical convergence of the method for non-convex optimization under a set of explicit assumptions. Our results indicate that the compressed method can match the convergence rate of the uncompressed one, making the proposed method potentially appealing for distributed processing of large datasets.