CRII: OAC: An Efficient Lossy Compression Framework for Reducing Memory Footprint for Extreme-Scale Deep Learning on GPU-Based HPC Systems

深度学习（DL）在科学探索、国家安全、智能环境、医疗保健等许多科学技术领域迅速发展成为最先进的技术。许多DL应用程序需要使用高性能计算（HPC）资源来处理大量数据。例如，研究人员和科学家正在HPC基础设施中使用极端尺度DL应用程序来分类极端天气模式和高能粒子。近年来，使用图形处理单元（gpu）来加速深度学习应用越来越受到人们的关注。然而，不断增长的深度学习应用规模给当今基于gpu的高性能计算基础设施带来了许多挑战。关键的挑战是内存需求与其在gpu上的可用性之间的巨大差距（例如，一到两个数量级）。该项目旨在通过开发一种新的框架来填补这一空白，通过极端规模DL应用的数据压缩技术有效地减少内存需求。拟议的研究将增强基于gpu的高性能计算基础设施在许多依赖DL技术的科学学科的广泛社区。该项目将连接机器学习和高性能计算社区，并增加它们之间的互动。教育和参与活动包括开发与数据压缩相关的新课程，指导一组选定的高中生参加为期一年的区域科学博览会研究项目，以及增加社区对利用高性能计算基础设施进行深度学习技术的理解。该项目还将鼓励学生对高性能计算环境下DL技术的相关研究感兴趣，并促进与多个国家实验室的研究合作。现有的用于训练极端规模深度神经网络（dnn）的最先进的GPU内存节省方法存在高性能开销和/或低内存占用减少的问题。错误有界的有损压缩是一种很有前途的方法，可以显著减少内存占用，同时仍然满足所需的分析精度。该项目将探索如何利用DNN中间数据的错误有界有损压缩来减少极端规模DNN训练的内存占用。该项目有三个阶段的研究计划。首先，该团队将全面研究将错误有界有损压缩应用于DNN中间数据对验证精度和训练性能的影响，在目标DNN和数据集上使用不同的错误有界有损压缩器、压缩模式和错误界限。其次，团队将基于影响分析结果对不同中间数据优化合适的误差有界有损压缩器的压缩质量，并设计一种有效的方案来自适应应用最适合的压缩方案。最后，该团队将在最先进的gpu的有损压缩框架上优化压缩性能。该团队将评估关于高分辨率气候分析和高能粒子物理应用的拟议框架，并将其与基于内存占用减少率和训练性能改进（例如，吞吐量、时间、历元数）的现有最先进技术进行比较。该项目将使科学家和研究人员能够使用给定的计算资源以快速有效的方式训练极端规模的深度神经网络，为新发现开辟机会。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Deep learning (DL) has rapidly evolved to a state-of-the-art technique in many science and technology disciplines, such as scientific exploration, national security, smart environment, and healthcare. Many of these DL applications require using high-performance computing (HPC) resources to process large amounts of data. Researchers and scientists, for instance, are employing extreme-scale DL applications in HPC infrastructures to classify extreme weather patterns and high-energy particles. In recent years, using Graphics Processing Units (GPUs) to accelerate DL applications has attracted increasing attention. However, the ever-increasing scales of DL applications bring many challenges to today’s GPU-based HPC infrastructures. The key challenge is the huge gap (e.g., one to two orders of magnitude) between the memory requirement and its availability on GPUs. This project aims to fill this gap by developing a novel framework to reduce the memory demand effectively and efficiently via data compression technologies for extreme-scale DL applications. The proposed research will enhance the GPU-based HPC infrastructures in broad communities for many scientific disciplines that rely on DL technologies. The project will connect machine learning and HPC communities and increase interactions between them. Educational and engagement activities include developing new curriculum related to data compression, mentoring a selected group of high school students in a year-long research project for a regional Science Fair competition, and increasing the community's understanding of leveraging HPC infrastructures for DL technologies. The project will also encourage student interest in research related to DL technologies on HPC environment and promote research collaborations with multiple national laboratories.Existing state-of-the-art GPU memory saving methods for training extreme-scale deep neural networks (DNNs) suffer from high performance overhead and/or low memory footprint reduction. Error-bounded lossy compression is a promising approach to significantly reduce the memory footprint while still meeting the required analysis accuracy. This project will explore how to leverage error-bounded lossy compression on DNN intermediate data to reduce the memory footprint for extreme-scale DNN training. The project has a three-stage research plan. First, the team will comprehensively investigate the impacts of applying error-bounded lossy compression to DNN intermediate data on both validation accuracy and training performance, using different error-bounded lossy compressors, compression modes, and error bounds on the targeted DNNs and datasets. Second, the team will optimize the compression quality of suitable error-bounded lossy compressors on different intermediate data based on the impact analysis outcome, and design an efficient scheme to adaptively apply a best-fit compression solution. Finally, the team will optimize the compression performance on the proposed lossy compression framework for state-of-the-art GPUs. The team will evaluate the proposed framework on high-resolution climate analytics and high-energy particle physics applications and compare it with existing state-of-the-art techniques based on both the memory footprint reduction ratio and training performance improvements (e.g., throughput, time, epoch number). The project will enable scientists and researchers to train extreme-scale DNNs with a given set of computing resources in a fast and efficient manner, opening opportunities for new discoveries.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.