On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learning
On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learning
复制标题
关于分布式深度学习压缩通信的理论分析与实际实现之间的差异
DOI：
--
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
Panos Kalnis
中科院分区：
文献类型：
--
作者：
Aritra Dutta;E. Bergou;A. Abdelmoniem;Chen;Atal Narayan Sahu;Marco Canini;Panos Kalnis
采用稀疏化或随机梯度量化形式的压缩通信来降低深度神经网络的分布式数据并行训练中的通信成本。然而，理论和实践之间存在着差距：虽然大多数现有压缩方法的理论分析假设压缩应用于整个模型的梯度，但许多实际实现分别对模型的每一层的梯度进行操作。在本文中，我们证明了逐层压缩在理论上更好，因为对于大范围的有偏和无偏压缩方法，收敛速率的上限是整个模型压缩的收敛速率的上限。然而，尽管有理论上的限制，我们对六种著名方法的实验研究表明，在实践中，收敛性可能会更好，也可能不会更好，这取决于实际的训练模型和压缩比。我们的研究结果表明，深度学习框架支持逐层和整个模型压缩是有利的。
Compressed communication, in the form of sparsification or quantization of stochastic gradients, is employed to reduce communication costs in distributed data-parallel training of deep neural networks. However, there exists a discrepancy between theory and practice: while theoretical analysis of most existing compression methods assumes compression is applied to the gradients of the entire model, many practical implementations operate individually on the gradients of each layer of the model.In this paper, we prove that layer-wise compression is, in theory, better, because the convergence rate is upper bounded by that of entire-model compression for a wide range of biased and unbiased compression methods. However, despite the theoretical bound, our experimental study of six well-known methods shows that convergence, in practice, may or may not be better, depending on the actual trained model and compression ratio. Our findings suggest that it would be advantageous for deep learning frameworks to include support for both layer-wise and entire-model compression.
影响因子：
2.9
作者：
Montavon, Gregoire;Samek, Wojciech;Mueller, Klaus-Robert
通讯作者：
Mueller, Klaus-Robert