CAREER: Dynamic Management of Compressed Arrays for High-Performance Computing Applications

高性能计算（HPC）使科学和工程的所有领域都取得了重大进展，使研究人员能够模拟在正常实验室环境中难以实现的复杂现象。随着新的HPC系统上线，计算速度远远超过数据移动速度。因此，数据移动可能会限制应用程序性能和系统吞吐量。然而，这种差异允许花费计算时间来降低HPC应用程序中移动数据的带宽要求，从而缓解性能瓶颈。该项目研究数据压缩和聚合技术的性能和实用性，以减少大规模科学应用程序通信，计算和存储的数据量。该项目的成果是一个变革性的数据管理运行时，它允许需要大量内存的科学和工程应用程序在更便宜，更通用的系统上运行，内存更少。因此，可以大大提高工作负载的吞吐量，促进其相关领域的研究进展。此外，减少应用程序所需的内存量允许应用程序运行更大和更详细的问题，允许科学家和工程师运行和分析以前棘手的实验。最后，这个项目旨在通过创建一个多学期的动手研究课程，扩大本科生的使用和理解HPC。本课程让STEM学生参与构建/设计/使用下一代HPC系统和应用程序，并为他们提供在校园研究机会，研究生院和现代劳动力中取得成功所需的跨学科技能。本项目通过添加逻辑来动态管理压缩数据，改进了当前最先进的有损和无损数据压缩;减少了高成本压缩和解压缩时间的性能影响。数据管理运行时允许应用程序用户/开发人员选择要注册的变量子集。从分配的角度来看，数据压缩驻留在主存储器中，并在所有进程间/进程内数据运动期间保持压缩。对于节点间通信，运行时在传输之前聚合具有相同目的地节点的消息。计算所需的数据在使用前被解压缩，并被放置在可重新配置的软件管理的缓存中，该缓存利用预取器在使用前对数据进行重新配置，从而限制了应用程序关键路径上的延迟。对于使用有损压缩的变量，运行库通过动态更改有损压缩误差界限，寻求减少超出应用程序可容忍范围的误差累积。该项目评估了一组不同的代理应用程序和生产级应用程序上的数据管理运行时，这些应用程序具有不同的内存需求、通信计算比率、通信模式，该项目由CCF部门软件和硬件基础计划和刺激竞争研究的既定计划（EPSCoR）联合资助。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

High-performance computing (HPC) has enabled significant advancements across all fields of science and engineering by allowing researchers to simulate complex phenomena that are difficult, if not impossible, in a normal laboratory setting. As new HPC systems come online, computational speed far exceeds the speed of data movement. Thus, data movement can limit application performance and system throughput. However, the disparity allows the expenditure of computational time to lower the bandwidth requirements to move data in HPC applications, mitigating performance bottlenecks. This project investigates the performance and utility of data compression and aggregation techniques to reduce the volume of data communicated, computed on, and stored by large-scale scientific applications. An outcome of this project is a transformative data management runtime that allows science and engineering applications that require large amounts of memory enables them to be run on cheaper and more common systems with less memory. Thus, the throughput of workloads can be greatly improved, facilitating research progress in their respected areas. Furthermore, reducing the amount of memory required by the application allows the application to run larger and more detailed problems, allowing scientists and engineers to run and analyze previously intractable experiments. Finally, this project seeks to broaden undergraduates' use and understanding of HPC by creating a multi-semester hands-on research course. This course engages STEM students to build/design/use the next generation of HPC systems and applications and prepares them with the cross-disciplinary skills needed to succeed in on-campus research opportunities, graduate school, and the modern workforce.This project improves current state-of-the-art lossy and lossless data compression by adding logic to dynamically manage compressed data; reducing the performance impact of high-cost compression and decompression times. The data management runtime allows application users/developers to select a subset of variables to register. Data, from the point of allocation, resides compressed in main memory and remains compressed during all inter/intra-process data motion. For inter-node communication, the runtime aggregates messages with the same destination node before transmission. Data that are needed for computation are decompressed just before use and are placed in a reconfigurable software-managed cache that utilizes a prefetcher to decompress data prior to use, limiting delays on the critical path of the application. For variables that use lossy compression, the runtime seeks to mitigate the accumulation of error beyond what the application can tolerate by dynamically altering the lossy compression error bound. This project evaluates the data management runtime on a diverse set of proxy application and production-level applications with varying memory requirements, communication computation ratios, communication patterns, and data access patterns.This project is jointly funded by CCF Division Software and Hardware Foundations Program and the Established Program to Stimulate Competitive Research (EPSCoR).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.