CSR: Small: CONCERT: Designing Scalable Communication Runtimes with On-the-fly Compression for HPC and AI Applications on Heterogeneous Architectures

在人工智能(AI)和高性能计算(HPC)具有巨大变革潜力的世界里，本研究的目标是开发一种创新的通信和压缩堆栈Concert，以释放异类体系结构的全部功能，并推动高性能和可扩展性。通过利用新兴的加速器和网络硬件，Concert寻求解决在利用异类架构、扩展通信和集成应用程序不可知的动态数据压缩方面的根本挑战。该项目的意义在于它有潜力通过高效利用异类资源来推动人工智能和高性能计算领域的发展，从而提高性能和可扩展性。音乐会的影响超出了科学进步的范畴。该项目将为设计和部署下一代HPC系统提供宝贵的指导方针，使学术界和工业界的用户受益。通过积极促进多样性和包容性，特别是在代表性不足的少数群体和女学生中，该项目促进了一个更具包容性的STEM环境。研究成果将有助于课程的改进，支持高性能计算、深度/机器学习和数据分析方面的教育和研究。此外，将结果传播给协作组织将对其HPC软件应用程序产生积极影响，使整个社会受益。在过去几年中，人工智能(AI)和高性能计算(HPC)应用程序通过利用现代HPC系统中高度异质硬件的最新趋势，不断提高性能。这些应用程序具有很高的通信要求，并且在集群有限的带宽下交换大量数据。然而，对于应用程序来说，高效地使用系统中的所有可用资源来利用新兴的即时压缩支持来扩展通信是具有挑战性的。为此，提出了一种自适应通信/压缩协议栈Constant(Scalable Communications Runtime with on-the-Fly Compression for HPC and AI Applications for HPC and AI Applications on-the-Fly Compression)。Concert通过负载感知和架构感知功能分区(FP)动态使用专用资源。它增强了使用消息传递接口(MPI)编程大型应用程序的现有事实上的标准。在本研究下需要关注的具体问题是：1)有效地支持异类硬件上的MPI/混合编程模型，以扩大通信和即时压缩；2)设计基于FP的方案来卸载通信/压缩任务；3)设计通信/压缩FP方案以支持来自数千个端点的放大请求；以及4)研究这些方案在性能和可扩展性方面的优势。拟议研究的变革性影响使广泛的人工智能和高性能计算应用程序能够高效、透明地利用来自多个供应商的新兴加速器和网络硬件。一个强大的软件分发和数据传播计划也被提出，以对学术和工业HPC/AI社区产生更广泛的影响。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

In a world where Artificial Intelligence (AI) and High-Performance Computing (HPC) hold immense potential for transformative advancements, this research aims to develop CONCERT, an innovative communication and compression stack, to unlock the full power of heterogeneous architectures and drive high performance and scalability. By leveraging emerging accelerators and networking hardware, CONCERT seeks to address fundamental challenges in utilizing heterogeneous architectures, scaling communication, and integrating application agnostic on-the-fly data compression. The project's significance lies in its potential to advance the field of AI and HPC by enabling efficient utilization of heterogeneous resources, resulting in enhanced performance and scalability. CONCERT's impact extends beyond scientific advancements. The project will provide valuable guidelines for designing and deploying next-generation HPC systems, benefiting users in academia and industry. By actively promoting diversity and inclusion, particularly among underrepresented minorities and female students, the project fosters a more inclusive STEM environment. The research outcomes will contribute to curriculum advancements, supporting education and research in HPC, Deep/Machine Learning, and Data Analytics. Additionally, the dissemination of results to collaborating organizations will positively impact their HPC software applications, benefiting society as a whole.Over the last few years, Artificial Intelligence (AI) and High-Performance Computing (HPC) applications have been continuously enhanced for performance by exploiting the latest trends in highly heterogeneous hardware in modern HPC systems. These applications have high communication requirements and exchange massive amounts of data given a cluster’s limited bandwidth. However, it is challenging for an application to efficiently use all resources available in the system to scale up communication with the emerging on-the-fly compression support. For this reason, an adaptive communication/compression stack called CONCERT (sCalable cOmmunicatioN Runtimes with On-the-fly Compression for HPC and AI Applications on hEterogenous aRchiTectures) is proposed. CONCERT dynamically employs dedicated resources through load and architectural aware Functional Partitioning (FP). It enhances the existing de-facto standard for programming large-scale applications using the Message Passing Interface (MPI). Specific issues to be focused under this research are: 1) Efficient support for MPI/Hybrid programming models on heterogeneous hardware to scale-up communication and on-the-fly compression, 2) Designing FP-based schemes to offload communication /compression tasks, 3) Designing a communication/compression FP scheme to support scale-up requests from thousands of endpoints, and 4) Studying the benefits of these schemes in terms of performance and scalability. The transformative impact of the proposed research enables a broad range of AI and HPC applications to efficiently and transparently leverage the emerging accelerators and networking hardware from multiple vendors. A strong software distribution and data dissemination plan is also proposed to have a broader impact on academic and industrial HPC/AI communities.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.