CRII: OAC: A Compressor-Assisted Collective Communication Framework for GPU-Based Large-Scale Deep Learning

由于更大的训练数据集、更大的神经网络模型和新的算法/技术，现代深度学习的规模迅速扩大。这对当前的分布式高性能计算（HPC）基础设施提出了重大挑战，因为更大规模的训练在节点之间传递更重要的梯度消息会导致更昂贵的集体通信成本。更强大的硬件平台不一定有助于克服这个性能瓶颈，因为需要优化的中间件支持来充分释放平台的计算能力。该项目旨在通过提供特定于梯度的有损压缩技术和优化的gpu感知压缩器辅助集体通信框架来缩小训练规模与基础设施能力之间的差距，从而系统地减少梯度消息大小并提高通信性能。交付物可以帮助最终用户在保持训练准确性的情况下获得更快的训练速度。这项研究的成功可以促进传统人工智能研究的进展，如计算机视觉和自然语言处理，以及新兴的人工智能科学研究领域科学，包括宇宙学，x射线成像和药物发现。该项目还通过利用研究成果开发新的课程和教学工具，在高性能计算和人工智能领域指导大学生和培训K-12学生，为教育和参与活动做出贡献。使用当前的集体通信库进行大规模分布式深度学习可能会产生显著的通信开销，因为梯度消息很大。对梯度消息应用有损压缩技术可以潜在地减少通信开销。然而，有几个重要的开放性研究问题需要研究以确保性能的提高：1)当前的有损压缩器对梯度数据是否足够有效？2)有损压缩器如何有效地集成到gpu感知的集体通信框架中？3) GPU资源如何在不同任务间高效共享？该项目解决了这些问题，并为大规模深度学习提供了一种新的压缩机辅助gpu感知集体通信框架。具体而言，该团队1)研究了使用错误有界科学数据有损压缩器压缩梯度数据的效率，并利用现有不同压缩器的优势开发了一种新的梯度压缩器，以获得更好的压缩比和训练精度；2)设计新型压缩器的GPU实现，并将其集成到支持GPU的MPI中，优化工作流程，最终将梯度压缩器的成本隐藏在通信成本中；3)分析了深度学习训练和压缩辅助集体通信的GPU资源利用率，设计了一种新的通信框架，在同一GPU上实现训练、压缩和集体计算（如约简）的任务调度，实现端到端深度学习训练的最优资源共享。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The scale of modern deep learning expands rapidly due to larger training datasets, larger neural network models, and new algorithms/techniques. It presents significant challenges to the current distributed high-performance computing (HPC) infrastructures since larger-scale training incurs more expensive collective communication costs for passing more significant gradient messages among nodes. A more powerful hardware platform may not necessarily help overcome this performance bottleneck, as optimized middleware supports are demanded to unleash the platform's computing capacity fully. This project aims to close the gap between the training scale and the infrastructure's capability by providing gradient-specific lossy compression techniques and an optimized GPU-aware compressor-assisted collective communication framework to reduce the gradient message sizes and improve communication performance systematically. The deliverables can help the end-users to get significantly faster training speed with preserved training accuracy. The success of this research can promote progress in both traditional AI research, such as computer vision and natural language processing, and emerging AI for Science research for domain sciences, including cosmology, X-ray imaging, and drug discovery. This project also contributes to educational and engagement activities by leveraging the research outcome to develop new curriculums and teaching tools for mentoring college students and training K-12 students in HPC and AI areas.Using current collective communication libraries for large-scale distributed deep learning can yield significant communication overhead since the gradient messages are large. Applying lossy compression techniques to gradient messages could potentially reduce the communication overhead. However, several important open research questions should be investigated to ensure the performance gain: 1) Are the current lossy compressors efficient enough for gradient data? 2) How can lossy compressors efficiently integrate into a GPU-aware collective communication framework? 3) How could the GPU resources be efficiently shared among different tasks? This project addresses these questions and delivers a novel compressor-assisted GPU-aware collective communication framework for large-scale deep learning. Specifically, the team 1) investigates the efficiency of using error-bounded scientific data lossy compressors to compress gradient data and develops a new gradient compressor by leveraging the advantages of different existing compressors to achieve a better compression ratio and training accuracy; 2) designs the new compressor's GPU implementation and integrates it into the GPU-aware MPI, then optimizes the workflow to ultimately hide the gradient compressor's cost in the communication cost; 3) profiles the GPU resource utilization of both the deep learning training and the compressor-assist collective communications, and designs a new communication framework to enable task scheduling of training, compression, and collectives' computations (e.g., reduction) on the same GPU to achieve optimal resource sharing for the end-to-end deep learning training.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.