A Feature-Driven Fixed-Ratio Lossy Compression Framework for Real-World Scientific Datasets
A Feature-Driven Fixed-Ratio Lossy Compression Framework for Real-World Scientific Datasets
复制标题
DOI：
10.1109/icde55515.2023.00116
复制
发表时间：
2023-04
期刊：
影响因子：
--
通讯作者：
Md. Hasanur Rahman;S. Di;Kai Zhao;Robert Underwood;Guanpeng Li;F. Cappello
中科院分区：
文献类型：
--
作者：
Md. Hasanur Rahman;S. Di;Kai Zhao;Robert Underwood;Guanpeng Li;F. Cappello
当今的科学应用和先进的仪器设备每天都在产生极其大量的数据，因此误差控制的有损压缩已成为科学数据存储和管理的关键技术。然而，现有的有损科学数据压缩器主要是基于错误控制驱动机制设计的，由于内存/存储容量和网络带宽等数据处理/管理资源的限制，无法有效地应用于需要达到理想压缩比的固定比例用例。为了解决这一差距，我们提出了一种低成本的与压缩机无关的特征驱动的固定比率有损压缩框架（FXRZ）。主要贡献有三方面。(1)基于广泛的应用数据集，我们对不同数据特征与压缩比之间的相关性进行了深入分析，这是我们框架的基础工作。(2)我们提出了一系列优化策略，使框架能够以非常低的计算成本达到相当高的识别期望误差配置的精度。(3)我们使用4个最先进的错误控制有损压缩器对来自不同领域的4个不同应用程序的10个不同快照和基于模拟配置的现实世界科学数据集进行了全面评估。我们的实验表明，FXRZ比目前最先进的相关工作高出108倍。在一台超级计算机上4096个核的实验表明，与整体并行数据转储的相关工作相比，性能提高了1.18 ~ 8.71倍。
Today’s scientific applications and advanced instruments are producing extremely large volumes of data everyday, so that error-controlled lossy compression has become a critical technique to the scientific data storage and management. Existing lossy scientific data compressors, however, are designed mainly based on error-control driven mechanism, which cannot be efficiently applied in the fixed-ratio use-case, where a desired compression ratio needs to be reached because of the restricted data processing/management resources such as limited memory/storage capacity and network bandwidth. To address this gap, we propose a low-cost compressor-agnostic feature-driven fixed-ratio lossy compression framework (FXRZ). The key contributions are three-fold. (1) We perform an in-depth analysis of the correlation between diverse data features and compression ratios based on a wide range of application datasets, which is a fundamental work for our framework. (2) We propose a series of optimization strategies that can enable the framework to reach a fairly high accuracy in identifying the expected error configuration with very low computational cost. (3) We comprehensively evaluate our framework using 4 state-of-the-art error-controlled lossy compressors on 10 different snapshots and simulation configuration-based real-world scientific datasets from 4 different applications across different domains. Our experiment shows that FXRZ outperforms the state-of-the-art related work by 108×. The experiments with 4,096 cores on a supercomputer show a performance gain of 1.18∼8.71× than the related work in overall parallel data dumping.