CAREER: Dynamic Management of Compressed Arrays for High-Performance Computing Applications

高性能计算（HPC）使研究人员能够模拟在普通实验室环境中很难（如果不是不可能的话）的复杂现象，从而在所有科学和工程领域取得了重大进展。随着新的高性能计算系统上线，计算速度远远超过数据移动的速度。因此，数据移动会限制应用程序性能和系统吞吐量。然而，这种差异使得计算时间的支出可以降低在HPC应用程序中移动数据的带宽需求，从而缓解性能瓶颈。该项目研究了数据压缩和聚合技术的性能和效用，以减少大规模科学应用程序通信、计算和存储的数据量。该项目的一个成果是一个变革性的数据管理运行时，它允许需要大量内存的科学和工程应用程序在更便宜、更通用、内存更少的系统上运行。因此，工作量的吞吐量可以大大提高，促进研究进展在他们所尊重的领域。此外，减少应用程序所需的内存使应用程序能够运行更大、更详细的问题，使科学家和工程师能够运行和分析以前难以处理的实验。最后，本计画透过建立一个多学期的动手研究课程，以扩大本科生对HPC的使用和理解。本课程吸引STEM学生构建/设计/使用下一代高性能计算系统和应用程序，并为他们在校园研究机会，研究生院和现代劳动力中取得成功所需的跨学科技能做好准备。该项目通过添加逻辑来动态管理压缩数据，改进了当前最先进的有损和无损数据压缩；减少高成本压缩和解压缩时间对性能的影响。数据管理运行时允许应用程序用户/开发人员选择要注册的变量子集。数据，从分配点开始，驻留在压缩的主存中，并在所有进程间/进程内数据移动期间保持压缩。对于节点间通信，运行时在传输前聚合具有相同目标节点的消息。计算所需的数据在使用之前被解压缩，并放置在一个可重构的软件管理的缓存中，该缓存利用预取器在使用之前解压缩数据，从而限制了应用程序关键路径上的延迟。对于使用有损压缩的变量，运行时试图通过动态更改有损压缩错误界限来减轻超出应用程序可容忍范围的错误积累。这个项目在一组不同的代理应用程序和生产级应用程序上评估数据管理运行时，这些应用程序具有不同的内存需求、通信计算比率、通信模式和数据访问模式。该项目由CCF部门软件和硬件基金会计划和建立计划共同资助，以刺激竞争研究（EPSCoR）。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

High-performance computing (HPC) has enabled significant advancements across all fields of science and engineering by allowing researchers to simulate complex phenomena that are difficult, if not impossible, in a normal laboratory setting. As new HPC systems come online, computational speed far exceeds the speed of data movement. Thus, data movement can limit application performance and system throughput. However, the disparity allows the expenditure of computational time to lower the bandwidth requirements to move data in HPC applications, mitigating performance bottlenecks. This project investigates the performance and utility of data compression and aggregation techniques to reduce the volume of data communicated, computed on, and stored by large-scale scientific applications. An outcome of this project is a transformative data management runtime that allows science and engineering applications that require large amounts of memory enables them to be run on cheaper and more common systems with less memory. Thus, the throughput of workloads can be greatly improved, facilitating research progress in their respected areas. Furthermore, reducing the amount of memory required by the application allows the application to run larger and more detailed problems, allowing scientists and engineers to run and analyze previously intractable experiments. Finally, this project seeks to broaden undergraduates' use and understanding of HPC by creating a multi-semester hands-on research course. This course engages STEM students to build/design/use the next generation of HPC systems and applications and prepares them with the cross-disciplinary skills needed to succeed in on-campus research opportunities, graduate school, and the modern workforce.This project improves current state-of-the-art lossy and lossless data compression by adding logic to dynamically manage compressed data; reducing the performance impact of high-cost compression and decompression times. The data management runtime allows application users/developers to select a subset of variables to register. Data, from the point of allocation, resides compressed in main memory and remains compressed during all inter/intra-process data motion. For inter-node communication, the runtime aggregates messages with the same destination node before transmission. Data that are needed for computation are decompressed just before use and are placed in a reconfigurable software-managed cache that utilizes a prefetcher to decompress data prior to use, limiting delays on the critical path of the application. For variables that use lossy compression, the runtime seeks to mitigate the accumulation of error beyond what the application can tolerate by dynamically altering the lossy compression error bound. This project evaluates the data management runtime on a diverse set of proxy application and production-level applications with varying memory requirements, communication computation ratios, communication patterns, and data access patterns.This project is jointly funded by CCF Division Software and Hardware Foundations Program and the Established Program to Stimulate Competitive Research (EPSCoR).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.