Learn Locally, Correct Globally: A Distributed Algorithm for Training Graph Neural Networks
Learn Locally, Correct Globally: A Distributed Algorithm for Training Graph Neural Networks
复制标题
DOI：
--
复制
发表时间：
2021-11
期刊：
影响因子：
--
通讯作者：
M. Ramezani;Weilin Cong;Mehrdad Mahdavi;M. Kandemir;A. Sivasubramaniam
中科院分区：
文献类型：
--
作者：
M. Ramezani;Weilin Cong;Mehrdad Mahdavi;M. Kandemir;A. Sivasubramaniam
尽管最近图神经网络(GNN)取得了成功，但在大型图上训练GNN仍然具有挑战性。现有服务器的有限资源容量、图中节点之间的依赖关系以及集中式存储和模型学习带来的隐私问题，促使人们需要设计一种有效的分布式GNN训练算法。然而，现有的分布式GNN训练方法要么存在过高的通信开销，要么存在较大的内存开销，阻碍了它们的可扩展性。为了克服这些问题，我们提出了一种通信高效的分布式GNN训练技术，命名为$\Text{{LearnalLearchGlobalRight}}$(LLCG)。为了减少通信和存储开销，LLCG中的每个本地机器首先通过忽略不同机器之间节点之间的依赖关系来对其本地数据训练一个GNN，然后将本地训练的模型发送到服务器进行周期性模型平均。但是，忽略节点依赖关系可能会导致性能显著下降。为了解决性能下降的问题，我们建议在服务器上应用$\Text{{全局服务器更正}}$来精炼本地学习的模型。我们严格分析了使用周期模型平均的分布式方法训练GNN的收敛情况，指出单纯地应用周期模型平均而忽略节点间的相关性将会产生不可约的残差。然而，这种残留误差可以通过利用所提出的全局校正来消除，从而获得快速的收敛速度。在真实数据集上的大量实验表明，LLCG可以在不损害性能的情况下显著提高效率。
Despite the recent success of Graph Neural Networks (GNNs), training GNNs on large graphs remains challenging. The limited resource capacities of the existing servers, the dependency between nodes in a graph, and the privacy concern due to the centralized storage and model learning have spurred the need to design an effective distributed algorithm for GNN training. However, existing distributed GNN training methods impose either excessive communication costs or large memory overheads that hinders their scalability. To overcome these issues, we propose a communication-efficient distributed GNN training technique named $\text{{Learn Locally, Correct Globally}}$ (LLCG). To reduce the communication and memory overhead, each local machine in LLCG first trains a GNN on its local data by ignoring the dependency between nodes among different machines, then sends the locally trained model to the server for periodic model averaging. However, ignoring node dependency could result in significant performance degradation. To solve the performance degradation, we propose to apply $\text{{Global Server Corrections}}$ on the server to refine the locally learned models. We rigorously analyze the convergence of distributed methods with periodic model averaging for training GNNs and show that naively applying periodic model averaging but ignoring the dependency between nodes will suffer from an irreducible residual error. However, this residual error can be eliminated by utilizing the proposed global corrections to entail fast convergence rate. Extensive experiments on real-world datasets show that LLCG can significantly improve the efficiency without hurting the performance.