CRII: OAC: A Compressor-Assisted Collective Communication Framework for GPU-Based Large-Scale Deep Learning

由于更大的训练数据集、更大的神经网络模型和新的算法/技术，现代深度学习的规模迅速扩大。这给当前的分布式高性能计算(HPC)基础设施带来了巨大的挑战，因为更大规模的训练会导致节点之间传递更多显着的梯度消息而产生更昂贵的集体通信成本。更强大的硬件平台不一定有助于克服这一性能瓶颈，因为需要优化的中间件支持来充分释放平台的计算能力。该项目旨在通过提供特定于梯度的有损压缩技术和优化的GPU感知的压缩器辅助的集体通信框架来缩小训练规模与基础设施能力之间的差距，以系统地减少梯度消息大小并提高通信性能。交付成果可以帮助最终用户在保持训练精度的情况下获得显著更快的训练速度。这项研究的成功可以促进传统的人工智能研究，如计算机视觉和自然语言处理，以及新兴的人工智能科学研究，包括宇宙学、X射线成像和药物发现。该项目还通过利用研究成果开发新的课程和教学工具来指导大学生并在HPC和人工智能领域培训K-12学生，从而为教育和参与活动做出贡献。使用现有的集体通信库进行大规模分布式深度学习可能会产生显著的通信开销，因为梯度消息很大。将有损压缩技术应用于梯度消息可能会降低通信开销。然而，为了确保性能的提高，需要研究几个重要的开放研究问题：1)现有的有损压缩器对梯度数据是否足够有效？2)有损压缩器如何有效地集成到一个GPU感知的集体通信框架中？3)如何在不同任务之间有效地共享GPU资源？该项目解决了这些问题，并提供了一种新颖的压缩机辅助的GPU感知大规模深度学习的集体通信框架。具体地说，团队1)研究了使用误差界科学数据有损压缩器压缩梯度数据的效率，并利用现有不同压缩器的优势开发了一种新的梯度压缩器，以获得更好的压缩比和训练精度；2)设计了新压缩器的GPU实现，并将其集成到GPU感知的MPI中，然后对工作流程进行优化，最终将梯度压缩器的成本隐藏在通信成本中；3)介绍深度学习培训和压缩机辅助集体通信的GPU资源利用情况，并设计新的通信框架，以支持在同一GPU上进行培训、压缩和集体计算(例如缩减)的任务调度，以实现端到端深度学习培训的最佳资源共享。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The scale of modern deep learning expands rapidly due to larger training datasets, larger neural network models, and new algorithms/techniques. It presents significant challenges to the current distributed high-performance computing (HPC) infrastructures since larger-scale training incurs more expensive collective communication costs for passing more significant gradient messages among nodes. A more powerful hardware platform may not necessarily help overcome this performance bottleneck, as optimized middleware supports are demanded to unleash the platform's computing capacity fully. This project aims to close the gap between the training scale and the infrastructure's capability by providing gradient-specific lossy compression techniques and an optimized GPU-aware compressor-assisted collective communication framework to reduce the gradient message sizes and improve communication performance systematically. The deliverables can help the end-users to get significantly faster training speed with preserved training accuracy. The success of this research can promote progress in both traditional AI research, such as computer vision and natural language processing, and emerging AI for Science research for domain sciences, including cosmology, X-ray imaging, and drug discovery. This project also contributes to educational and engagement activities by leveraging the research outcome to develop new curriculums and teaching tools for mentoring college students and training K-12 students in HPC and AI areas.Using current collective communication libraries for large-scale distributed deep learning can yield significant communication overhead since the gradient messages are large. Applying lossy compression techniques to gradient messages could potentially reduce the communication overhead. However, several important open research questions should be investigated to ensure the performance gain: 1) Are the current lossy compressors efficient enough for gradient data? 2) How can lossy compressors efficiently integrate into a GPU-aware collective communication framework? 3) How could the GPU resources be efficiently shared among different tasks? This project addresses these questions and delivers a novel compressor-assisted GPU-aware collective communication framework for large-scale deep learning. Specifically, the team 1) investigates the efficiency of using error-bounded scientific data lossy compressors to compress gradient data and develops a new gradient compressor by leveraging the advantages of different existing compressors to achieve a better compression ratio and training accuracy; 2) designs the new compressor's GPU implementation and integrates it into the GPU-aware MPI, then optimizes the workflow to ultimately hide the gradient compressor's cost in the communication cost; 3) profiles the GPU resource utilization of both the deep learning training and the compressor-assist collective communications, and designs a new communication framework to enable task scheduling of training, compression, and collectives' computations (e.g., reduction) on the same GPU to achieve optimal resource sharing for the end-to-end deep learning training.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.