SPARQ-SGD: Event-Triggered and Compressed Communication in Decentralized Optimization
SPARQ-SGD: Event-Triggered and Compressed Communication in Decentralized Optimization
复制标题
DOI：
10.1109/tac.2022.3145576
复制
发表时间：
2023-02
影响因子：
6.8
通讯作者：
Navjot Singh;Deepesh Data;Jemin George;S. Diggavi
中科院分区：
文献类型：
--
作者：
Navjot Singh;Deepesh Data;Jemin George;S. Diggavi
在这篇文章中，我们提出并分析了SParsified Action Regulated Quantized-Stochastic Gradient Descent（SPARQ-SGD），这是一种通信效率高的算法，用于在具有$n$个节点的图上分散训练大规模机器学习模型，其中通信效率是通过压缩相邻节点之间的局部模型参数交换来实现的，只有当满足事件（局部可计算条件）时才会触发。具体来说，在SPARQ-SGD中，每个节点采取固定数量的局部梯度步骤，然后检查模型参数与其上次更新相比是否发生了显着变化;只有当变化超过某个阈值（由设计标准指定）时，它才使用量化和稀疏化压缩其局部模型参数，并将其传递给邻居。我们证明了SPARQ-SGD收敛为$O（\frac{1}{nT}）$和$O（\frac{1}{\sqrt{nT}}）$的强凸和非凸设置，分别匹配平原分散SGD的收敛速度。这表明，我们通过积极的压缩，本地迭代和事件触发的通信基本上免费获得了通信效率。我们在真实的数据集上评估SPARQ-SGD，以证明在实现类似性能的同时，在最先进的通信中节省了大量的费用。
In this article, we propose and analyze SParsified Action Regulated Quantized–Stochastic Gradient Descent (SPARQ-SGD), a communication-efficient algorithm for decentralized training of large-scale machine learning models over a graph with $n$ nodes, where communication efficiency is achieved using compressed exchange of local model parameters among neighboring nodes, which is triggered only when an event (a locally computable condition) is satisfied. Specifically, in SPARQ-SGD, each node takes a fixed number of local gradient steps and then checks if the model parameters have significantly changed compared to its last update; only when the change is beyond a certain threshold (specified by a design criterion), it compresses its local model parameters using both quantization and sparsification and communicates them to its neighbors. We prove that SPARQ-SGD converges as $O(\frac{1}{nT})$ and $O(\frac{1}{\sqrt{nT}})$ in the strongly convex and nonconvex settings, respectively, matching the convergence rates of plain decentralized SGD. This demonstrates that we get communication efficiency achieved by aggressive compression, local iterations, and event-triggered communication essentially for free. We evaluate SPARQ-SGD over real datasets to demonstrate significant amount of savings in communication over the state-of-the-art while achieving similar performance.