Toward Quantized Model Parallelism for Graph-Augmented MLPs Based on Gradient-Free ADMM Framework
Toward Quantized Model Parallelism for Graph-Augmented MLPs Based on Gradient-Free ADMM Framework
复制标题
DOI：
10.1109/tnnls.2022.3223879
复制
发表时间：
2022-12
影响因子：
10.4
通讯作者：
Junxiang Wang;Hongyi Li;Zheng Chai;Yongchao Wang;Yue Cheng;Liang Zhao
中科院分区：
文献类型：
--
作者：
Junxiang Wang;Hongyi Li;Zheng Chai;Yongchao Wang;Yue Cheng;Liang Zhao
尽管图形神经网络（GNN）在深度学习社区中很受欢迎，但它们遭受了几个挑战，包括过度平滑，过度阵型和梯度消失。最近，一系列模型试图通过首先增强节点功能，然后基于多层perceptron（MLP）施加节点函数来缓解这些问题，这些功能被广泛称为图形增强的MLP（GA-MLP）模型。但是，尽管GA-MLP型号享有更深层次的体系结构，以提高准确性，但它们的效率在很大程度上会恶化。此外，由于样本（即节点）在图中的依赖性（即节点）之间的依赖性，因此无法有效地应用流行的加速技术，例如随机反变或数据并行性。为了解决这些问题，在本文中，我们提出了一个并行图深度学习交流方向方法（PDADMM-G）框架以实现模型并行性：可以在GA-MLP模型中的每一层中的参数进行更新平行线。扩展的PDADMM-G-Q算法通过引入量化技术来降低通信成本。在PDADMM-G算法和PDADMM-G-Q算法的（量化）固定点的理论收敛中，有sublinear收敛速率$ o（1/k）$，其中$ k $是迭代的数量。广泛的实验证明了两种提出的算法的收敛性。此外，与九个基准数据集中的所有最新比较方法相比，它们导致更大的加速和更好的性能。最后但并非最不重要的一点是，拟议的PDADMM-G-Q算法将开销降低了45％，而不会丧失绩效。我们的代码可在https://github.com/xianggebenben/pdadmm-g上找到。
While graph neural networks (GNNs) are popular in the deep learning community, they suffer from several challenges including over-smoothing, over-squashing, and gradient vanishing. Recently, a series of models have attempted to relieve these issues by first augmenting the node features and then imposing node-wise functions based on multilayer perceptron (MLP), which are widely referred to as graph-augmented MLP (GA-MLP) models. However, while GA-MLP models enjoy deeper architectures for better accuracy, their efficiency largely deteriorates. Moreover, popular acceleration techniques such as stochastic-version or data-parallelism cannot be effectively applied due to the dependency among samples (i.e., nodes) in graphs. To address these issues, in this article, instead of data parallelism, we propose a parallel graph deep learning Alternating Direction Method of Multipliers (pdADMM-G) framework to achieve model parallelism: parameters in each layer of GA-MLP models can be updated in parallel. The extended pdADMM-G-Q algorithm reduces communication costs by introducing the quantization technique. Theoretical convergence to a (quantized) stationary point of the pdADMM-G algorithm and the pdADMM-G-Q algorithm is provided with a sublinear convergence rate $o(1/k)$ , where $k$ is the number of iterations. Extensive experiments demonstrate the convergence of two proposed algorithms. Moreover, they lead to a more massive speedup and better performance than all state-of-the-art comparison methods on nine benchmark datasets. Last but not least, the proposed pdADMM-G-Q algorithm reduces communication overheads by up to 45% without loss of performance. Our code is available at https://github.com/xianggebenben/pdADMM-G.