Efficient Encoding and Reconstruction of HPC Datasets for Checkpoint/Restart
Efficient Encoding and Reconstruction of HPC Datasets for Checkpoint/Restart
复制标题
DOI：
10.1109/msst.2019.00-14
复制
发表时间：
2019-05
期刊：
影响因子：
--
通讯作者：
Jialing Zhang;Xiaoyan Zhuo;Aekyeung Moon;Hang Liu;S. Son
中科院分区：
文献类型：
--
作者：
Jialing Zhang;Xiaoyan Zhuo;Aekyeung Moon;Hang Liu;S. Son
随着HPC应用程序产生的数据量达到EB范围，通常采用压缩技术来减少检查点时间和数量。由于无损技术在实现可观的数据减少方面的能力有限，因此有损压缩成为一个更好的选择。在这项工作中，提出了一种高效率的编码，专门建立的错误控制和高压缩比的有损压缩技术。具体来说，我们应用离散余弦变换与一种新的块分解策略直接双精度浮点数据集，而不是流行的基于预测的技术。此外，我们设计了一个自适应量化与两个特定的面向任务的量化器：保证误差界和更高的压缩比。使用真实世界的HPC数据集，我们的方法实现了3x-38x的压缩比，同时保证了指定的误差范围，表现出与最先进的有损压缩方法SZ和ZFP相当的性能。此外，我们的方法提供了可行的重建数据的各种检查点/重新启动的情况下，在FLASH应用程序，因此被认为是一个有前途的方法，在HPC I/O软件栈的有损数据压缩。
As the amount of data produced by HPC applications reaches the exabyte range, compression techniques are often adopted to reduce the checkpoint time and volume. Since lossless techniques are limited in their ability to achieve appreciable data reduction, lossy compression becomes a preferable option. In this work, a lossy compression technique with highly efficient encoding, purpose-built error control, and high compression ratios is proposed. Specifically, we apply a discrete cosine transform with a novel block decomposition strategy directly to double-precision floating point datasets instead of prevailing prediction-based techniques. Further, we design an adaptive quantization with two specific task-oriented quantizers: guaranteed error bounds and higher compression ratios. Using real-world HPC datasets, our approach achieves 3x-38x compression ratios while guaranteeing specified error bounds, showing comparable performance with state-of-the-art lossy compression methods, SZ and ZFP. Moreover, our method provides viable reconstructed data for various checkpoint/restart scenarios in the FLASH application, thus is considered to be a promising approach for lossy data compression in HPC I/O software stacks.