Training Recommender Systems at Scale: Communication-Efficient Model and Data Parallelism
Training Recommender Systems at Scale: Communication-Efficient Model and Data Parallelism
复制标题
DOI：
10.1145/3447548.3467080
复制
发表时间：
2020-10
期刊：
影响因子：
--
通讯作者：
Vipul Gupta;Dhruv Choudhary;P. T. P. Tang-P.-T.-P.-Tang-144669504;Xiaohan Wei;Xing Wang;Yuzhen Huang;A. Kejariwal;K. Ramc
中科院分区：
文献类型：
--
作者：
Vipul Gupta;Dhruv Choudhary;P. T. P. Tang-P.-T.-P.-Tang-144669504;Xiaohan Wei;Xing Wang;Yuzhen Huang;A. Kejariwal;K. Ramc
在本文中，我们认为混合并行--一种范式，采用数据并行主义（DP）和模型并行主义（MP）-规模的大型推荐模型的分布式训练。我们提出了一个称为动态通信保持（DCT）的压缩框架，用于通信高效的混合训练。DCT通过一个简单的硬阈值函数过滤网络中要通信的实体，只允许最相关的信息通过。对于通信高效的DP，DCT压缩在模型同步期间发送到参数服务器的参数梯度。阈值每几千次迭代仅更新一次，以减少压缩的计算开销。对于通信高效的MP，DCT结合了一种新的技术来分别压缩在前向和后向传播期间通过网络发送的激活和梯度。这是通过为数据中的每个训练样本识别和更新神经网络中最相关的神经元来完成的。我们在公开的自然语言处理和推荐模型和数据集上评估DCT，以及Facebook在生产中使用的推荐系统。DCT在DP和MP期间分别将通信减少至少100倍和20倍。该算法已在生产中部署，它将最先进的工业推荐模型的端到端训练时间提高了37%，而性能没有任何损失。
In this paper, we consider hybrid parallelism---a paradigm that employs both Data Parallelism (DP) and Model Parallelism (MP)---to scale distributed training of large recommendation models. We propose a compression framework called Dynamic Communication Thresholding (DCT) for communication-efficient hybrid training. DCT filters the entities to be communicated across the network through a simple hard-thresholding function, allowing only the most relevant information to pass through. For communication efficient DP, DCT compresses the parameter gradients sent to the parameter server during model synchronization. The threshold is updated only once every few thousand iterations to reduce the computational overhead of compression. For communication efficient MP, DCT incorporates a novel technique to compress the activations and gradients sent across the network during the forward and backward propagation, respectively. This is done by identifying and updating only the most relevant neurons of the neural network for each training sample in the data. We evaluate DCT on publicly available natural language processing and recommender models and datasets, as well as recommendation systems used in production at Facebook. DCT reduces communication by at least 100x and 20x during DP and MP, respectively. The algorithm has been deployed in production, and it improves end-to-end training time for a state-of-the-art industrial recommender model by 37%, without any loss in performance.