CAREER: Dynamic Management of Compressed Arrays for High-Performance Computing Applications

高性能计算(HPC)使研究人员能够模拟复杂的现象，这些现象在正常的实验室环境下即使不是不可能的，也是困难的，从而使科学和工程的所有领域都取得了重大进展。随着新的HPC系统上线，计算速度远远超过数据移动的速度。因此，数据移动可能会限制应用程序性能和系统吞吐量。然而，这种差异允许花费计算时间来降低在HPC应用程序中移动数据的带宽要求，从而缓解性能瓶颈。该项目调查数据压缩和聚合技术的性能和实用性，以减少大规模科学应用程序通信、计算和存储的数据量。该项目的一个成果是一个变革性的数据管理运行时，它允许需要大量内存的科学和工程应用程序在更便宜、更常见、内存更少的系统上运行。因此，可以极大地提高工作负荷的吞吐量，促进其所关注领域的研究进展。此外，减少应用程序所需的内存量允许应用程序运行更大、更详细的问题，从而允许科学家和工程师运行和分析以前难以处理的实验。最后，这个项目试图通过创建一个多学期的实践研究课程来扩大本科生对HPC的使用和理解。本课程让STEM学生构建/设计/使用下一代HPC系统和应用程序，并为他们准备好在校园研究机会、研究生院和现代工作人员中取得成功所需的跨学科技能。该项目通过添加动态管理压缩数据的逻辑来改进当前最先进的有损和无损数据压缩；减少高成本压缩和解压缩对性能的影响。数据管理运行时允许应用程序用户/开发人员选择要注册的变量子集。从分配点来看，数据驻留在主存储器中压缩，并且在所有进程间/进程内数据移动期间保持压缩状态。对于节点间通信，运行库在传输之前聚合具有相同目标节点的消息。计算所需的数据在使用前被解压缩，并被放置在可重新配置的软件管理的高速缓存中，该高速缓存在使用之前利用预取器来解压缩数据，从而限制了应用程序关键路径上的延迟。对于使用有损压缩的变量，运行库试图通过动态更改有损压缩误差界来缓解应用程序所能容忍的错误累积。该项目评估各种代理应用程序和生产级应用程序上的数据管理运行时，这些应用程序具有不同的内存要求、通信计算比率、通信模式和数据访问模式。该项目由CCF部门软件和硬件基础计划和已建立的激励竞争研究计划(EPSCoR)联合资助。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

High-performance computing (HPC) has enabled significant advancements across all fields of science and engineering by allowing researchers to simulate complex phenomena that are difficult, if not impossible, in a normal laboratory setting. As new HPC systems come online, computational speed far exceeds the speed of data movement. Thus, data movement can limit application performance and system throughput. However, the disparity allows the expenditure of computational time to lower the bandwidth requirements to move data in HPC applications, mitigating performance bottlenecks. This project investigates the performance and utility of data compression and aggregation techniques to reduce the volume of data communicated, computed on, and stored by large-scale scientific applications. An outcome of this project is a transformative data management runtime that allows science and engineering applications that require large amounts of memory enables them to be run on cheaper and more common systems with less memory. Thus, the throughput of workloads can be greatly improved, facilitating research progress in their respected areas. Furthermore, reducing the amount of memory required by the application allows the application to run larger and more detailed problems, allowing scientists and engineers to run and analyze previously intractable experiments. Finally, this project seeks to broaden undergraduates' use and understanding of HPC by creating a multi-semester hands-on research course. This course engages STEM students to build/design/use the next generation of HPC systems and applications and prepares them with the cross-disciplinary skills needed to succeed in on-campus research opportunities, graduate school, and the modern workforce.This project improves current state-of-the-art lossy and lossless data compression by adding logic to dynamically manage compressed data; reducing the performance impact of high-cost compression and decompression times. The data management runtime allows application users/developers to select a subset of variables to register. Data, from the point of allocation, resides compressed in main memory and remains compressed during all inter/intra-process data motion. For inter-node communication, the runtime aggregates messages with the same destination node before transmission. Data that are needed for computation are decompressed just before use and are placed in a reconfigurable software-managed cache that utilizes a prefetcher to decompress data prior to use, limiting delays on the critical path of the application. For variables that use lossy compression, the runtime seeks to mitigate the accumulation of error beyond what the application can tolerate by dynamically altering the lossy compression error bound. This project evaluates the data management runtime on a diverse set of proxy application and production-level applications with varying memory requirements, communication computation ratios, communication patterns, and data access patterns.This project is jointly funded by CCF Division Software and Hardware Foundations Program and the Established Program to Stimulate Competitive Research (EPSCoR).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.