Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
复制标题
DOI：
--
复制
发表时间：
2018-08
期刊：
影响因子：
--
通讯作者：
Jianyu Wang;Gauri Joshi
中科院分区：
文献类型：
--
作者：
Jianyu Wang;Gauri Joshi
通信高效的SGD算法允许节点执行本地更新并定期同步本地模型，在提高分布式SGD的速度和可扩展性方面非常有效。然而，对不同的交际减少策略进行严格的趋同分析和比较研究在很大程度上仍然是一个悬而未决的问题。本文提出了一种称为协作式SGD的统一框架，它包含了现有的通信效率较高的SGD算法，如周期平均、弹性平均和分散SGD。通过对协作式SGD算法的分析，为现有算法提供了新的收敛保证。此外，该框架使我们能够设计新的通信效率高的SGD算法，在减少通信开销和以低错误平台实现快速错误收敛之间取得最佳平衡。
Communication-efficient SGD algorithms, which allow nodes to perform local updates and periodically synchronize local models, are highly effective in improving the speed and scalability of distributed SGD. However, a rigorous convergence analysis and comparative study of different communication-reduction strategies remains a largely open problem. This paper presents a unified framework called Cooperative SGD that subsumes existing communication-efficient SGD algorithms such as periodic-averaging, elastic-averaging and decentralized SGD. By analyzing Cooperative SGD, we provide novel convergence guarantees for existing algorithms. Moreover, this framework enables us to design new communication-efficient SGD algorithms that strike the best balance between reducing communication overhead and achieving fast error convergence with low error floor.