CNS Core: Small: Transparently Scaling Graph Neural Network Training to Large-Scale Models and Graphs

具有数十亿条边的大规模图在许多工业、科学和工程领域中普遍存在，如推荐系统、社交图分析、知识库、材料科学和生物学。特别是，图神经网络(GNN)作为一种新兴的机器学习(ML)模型，由于其在许多任务中的优越性能而被越来越多地采用。不幸的是，由于缺乏对ML实践者的充分系统支持，在大规模真实世界图表上培训GNN的进展受到了破坏。该项目将发展对算法、系统和基础设施的基础研究，以满足对GNN培训系统的迫切和不断增长的需求，该系统可以扩展到对用户透明的大型图形数据集和大型可表达GNN模型。首先，该项目将开发拆分并行，这是一种新的并行训练范式，旨在通过向外扩展到分布式和多图形处理单元(GPU)系统来支持任意大规模的图形和GNN模型。分裂并行是针对GNN的特定瓶颈而量身定做的，并引入了一套技术来透明地在GPU之间分割训练计算。其次，该项目将开发可伸缩图形采样系统，这可能是GNN培训中的一个主要性能瓶颈。它将开发一种新的基于片段的GPU内采样方法，该方法透明地将样本拆分成多个片段，以最大限度地提高数据访问的局部性和可伸缩性。支持大规模图形和GNN模型将在广泛的领域引发创新，使ML从业者更容易开发大型且具有表现力的模型，而不必绕过当前GNN培训系统的可扩展性限制。该项目将开发并行训练和采样的新方法，并将在一般机器学习和图形分析领域引入算法、基础设施和系统设计方面的创新。该项目将强调技术转让，将研究结果纳入流行的开源GNN培训工具，如深度图库(DGL)。PI还将支持他们部门的同事使用知识图谱进行问题回答。该项目将改善研究生和本科生的培训，强调人口结构的多样性。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Large-scale graphs with billions of edges are ubiquitous in many industry, science, and engineering fields such as recommendation systems, social graph analysis, knowledge bases, materials science, and biology. In particular, Graph Neural Networks (GNN), an emerging class of machine learning (ML) models, are increasingly adopted due to their superior performance in many tasks. Unfortunately, the progress towards training GNNs on large-scale real-world graphs is undermined by the lack of adequate systems support for ML practitioners. This project will develop fundamental research on algorithms, systems, and infrastructures to meet the pressing and growing need for GNN training systems that can scale to both large graph datasets and large expressive GNN models transparently to users. First, this project will develop split parallelism, a novel parallel training paradigm designed to support arbitrarily large-scale graphs and GNN models by scaling out to distributed and multi-GPU (graphics processing unit) systems. Split parallelism is tailored to the specific bottlenecks of GNNs and introduces a set of techniques to transparently split the training computation across GPUs. Second, this project will develop systems for scalable graph sampling, which can be a major performance bottleneck in GNN training. It will develop a novel fragment-based in-GPU sampling approach that transparently splits samples into multiple fragments to maximize data access locality and scalability.Supporting large-scale graphs and GNN models will unleash innovation in a wide range of domains by making it easier for ML practitioners to develop large and expressive models without having to work around the scalability limitations of current GNN training systems. The project will develop novel approaches for parallel training and sampling and will introduce innovations in algorithms, infrastructure, and system design for the areas of general machine learning and graph analytics. This project will stress technology transfer to integrate the findings into popular open-source GNN training tools such as the Deep Graph Library (DGL). The PIs will also support colleagues at their department working on question answering using knowledge graphs. The project will improve the training of both graduate and undergraduate students, emphasizing demographic diversity.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.