An Algorithmic and Software Pipeline for Very Large Scale Scientific Data Compression with Error Guarantees
An Algorithmic and Software Pipeline for Very Large Scale Scientific Data Compression with Error Guarantees
复制标题
用于具有错误保证的超大规模科学数据压缩的算法和软件管道
DOI：
10.1109/hipc56025.2022.00039
复制
发表时间：
2022
期刊：
影响因子：
--
通讯作者：
Sanjay Ranka
中科院分区：
文献类型：
--
作者：
Tania Banerjee;J. Choi;Jaemoon Lee;Qian Gong;Ruonan Wang;S. Klasky;A. Rangarajan;Sanjay Ranka
对于存储科学数据而言，高效的数据压缩正变得日益关键，因为许多科学应用会产生大量数据。本文提出了一种用于数据压缩的端到端算法和软件流水线，它能保证原始数据（PD）和派生数据（即感兴趣的量，QoI）的误差界限。我们通过压缩由大规模聚变代码XGC生成的聚变数据来证明该流水线的有效性，XGC在一天内会产生数十拍字节的数据。我们证明了压缩是通过预留被称为暂存节点的计算资源来进行的，并且不会影响模拟性能。为了实现高效的并行I/O，该流水线使用了ADIOS2，许多代码（如XGC）已经将其用于并行I/O。我们表明，我们的方法可以将数据压缩两个数量级，同时保证PD和QoI的高精度。此外，压缩所需的资源量是模拟所需资源的百分之几，同时确保每个阶段的压缩时间小于相应的模拟时间。
该流水线由三个主要步骤组成。第一步使用区域分解将数据分解为小的子区域。然后每个子区域被独立压缩以实现高度的并行性。第二步使用现有技术来保证每个子区域原始数据的误差界限。第三步使用一种基于拉格朗日乘子的后处理优化技术来减少与每个子区域对应的数据的QoI误差。生成的拉格朗日乘子可以进一步量化或截断以提高压缩级别。我们方法的所有上述特点使其在保证对科学家至关重要的QoI误差的同时，非常实用地应用即时压缩。
Efficient data compression is becoming increasingly critical for storing scientific data because many scientific applications produce vast amounts of data. This paper presents an end-to-end algorithmic and software pipeline for data compression that guarantees both error bounds on primary data (PD) and derived data, known as Quantities of Interest (QoI).We demonstrate the effectiveness of the pipeline by compressing fusion data generated by a large-scale fusion code, XGC, which produces tens of petabytes of data in a single day. We demonstrate that the compression is conducted by setting aside computational resources known as staging nodes, and does not impact the simulation performance. For efficient parallel I/O, the pipeline uses ADIOS2, which many codes such as XGC already use for their parallel I/O. We show that our approach can compress the data by two orders of magnitude while guaranteeing high accuracy on both the PD and the QoIs. Further, the amount of resources required by compression is a few percent of the resources required by simulation while ensuring that the compression time for each stage is less than the corresponding simulation time.This pipeline consists of three main steps. The first step decomposes the data using domain decomposition into small subdomains. Each subdomain is then compressed independently to achieve a high level of parallelism. The second step uses existing techniques that guarantee error bounds on the primary data for each subdomain. The third step uses a post-processing optimization technique based on Lagrange multipliers to reduce the QoI errors for data corresponding to each subdomain. The Lagrange multipliers generated can be further quantized or truncated to increase the compression level. All of the above characteristics of our approach make it highly practical to apply on-the-fly compression while guaranteeing errors on QoIs that are critical to the scientists.