EventGraD: Event-Triggered Communication in Parallel Stochastic Gradient Descent
EventGraD: Event-Triggered Communication in Parallel Stochastic Gradient Descent
复制标题
DOI：
10.1109/mlhpcai4s51975.2020.00008
复制
发表时间：
2020-11
期刊：
影响因子：
--
通讯作者：
Soumyadip Ghosh;V. Gupta
中科院分区：
文献类型：
--
作者：
Soumyadip Ghosh;V. Gupta
并行系统中的通信消耗了大量的时间和能量，这往往是分布式机器学习的瓶颈。在本文中，我们提出了EventGraD -一个算法与事件触发的通信并行随机梯度下降。该算法的主要思想是修改通信的要求，在每个时期，只有在某些时期的必要时，通信。特别地，仅在参数值的变化超过阈值的情况下才传送参数。基于参数的变化率自适应地选择参数的阈值。自适应阈值保证了算法可以应用于不同数据集上的不同模型而不做任何改变。我们专注于用于训练MNIST数据集的流行卷积神经网络的数据并行训练，并表明EventGraD可以将通信负载减少高达70%，同时保持相同的准确度。
Communication in parallel systems consumes significant amount of time and energy which often turns out to be a bottleneck in distributed machine learning. In this paper, we present EventGraD - an algorithm with event-triggered communication in parallel stochastic gradient descent. The main idea of this algorithm is to modify the requirement of communication at every epoch to communicating only in certain epochs when necessary. In particular, the parameters are communicated only in the event when the change in their values exceed a threshold. The threshold for a parameter is chosen adaptively based on the rate of change of the parameter. The adaptive threshold ensures that the algorithm can be applied to different models on different datasets without any change. We focus on data-parallel training of a popular convolutional neural network used for training the MNIST dataset and show that EventGraD can reduce the communication load by up to 70% while retaining the same level of accuracy.