Evaluation of lossless and lossy algorithms for the compression of scientific datasets in netCDF-4 or HDF5 files
Evaluation of lossless and lossy algorithms for the compression of scientific datasets in netCDF-4 or HDF5 files
复制标题
DOI：
10.5194/gmd-12-4099-2019
复制
发表时间：
2019-09-23
影响因子：
5.1
通讯作者：
Gouillon, Flavien
中科院分区：
文献类型：
--
作者：
Delaunay, Xavier;Courtois, Aurelie;Gouillon, Flavien
随着科学数据集数量的不断增加，需要使用压缩技术来降低数据存储和传输成本，特别是对地观测任务地面段生成的海洋或气象数据集。这些数据大多以netCDF文件的形式产生。事实上，netdf -4/HDF5文件格式在全球科学界广泛使用，因为它们提供了有用的特性。HDF5特别提供了一个动态加载的过滤器插件，这样用户就可以编写压缩/解压缩过滤器，并在读取或写入数据到磁盘之前处理它们。本研究通过netCDF-4和HDF5工具在分析和真实科学浮点数据集上评估有损和无损压缩/解压缩方法。我们还介绍了数字舍入算法，这是一种受比特修饰算法启发的新的相对误差有界数据约简方法。数字舍入算法提供了高压缩比，同时在数据集中保持给定数量的有效数字。在压缩速度稍低的情况下，实现了比Bit Grooming算法更高的压缩比。
The increasing volume of scientific datasets requires the use of compression to reduce data storage and transmission costs, especially for the oceanographic or meteorological datasets generated by Earth observation mission ground segments. These data are mostly produced in netCDF files. Indeed, the netCDF-4/HDF5 file formats are widely used throughout the global scientific community because of the useful features they offer. HDF5 in particular offers a dynamically loaded filter plugin so that users can write compression/decompression filters, for example, and process the data before reading or writing them to disk. This study evaluates lossy and lossless compression/decompression methods through netCDF-4 and HDF5 tools on analytical and real scientific floating-point datasets. We also introduce the Digit Rounding algorithm, a new relative error-bounded data reduction method inspired by the Bit Grooming algorithm. The Digit Rounding algorithm offers a high compression ratio while keeping a given number of significant digits in the dataset. It achieves a higher compression ratio than the Bit Grooming algorithm with slightly lower compression speed.