Parallel I/O on Compressed Data Files: Semantics, Algorithms, and Performance Evaluation
Parallel I/O on Compressed Data Files: Semantics, Algorithms, and Performance Evaluation
复制标题
DOI：
10.1109/ccgrid49817.2020.00-74
复制
发表时间：
2020-05
期刊：
影响因子：
--
通讯作者：
S. Singh;E. Gabriel
中科院分区：
文献类型：
--
作者：
S. Singh;E. Gabriel
许多科学应用程序都是在数百兆字节甚至兆兆字节的数据集上运行的。大型数据集通常使用压缩来减小文件的大小。然而，到目前为止，并行I/O库不支持阅读和写压缩文件，在模拟之前/之后需要昂贵的顺序压缩/解压缩操作，或者省略并行I/O库的高级功能，例如集体I/O操作。本文介绍了压缩数据文件的并行I/O，讨论了在MPI I/O中支持压缩数据文件的关键挑战、需求和解决方案，以及使用压缩数据文件时某些MPI I/O操作的限制。本文详细介绍了压缩数据文件的个人读写操作的处理，并提出了一个扩展的两阶段集体I/O算法，以支持数据压缩。本文进一步提出并评估了基于Snappy压缩库和OMPIO并行I/O框架的实现。使用多个数据集进行的性能评估表明，在并行BeeGFS文件系统上使用数据压缩时，可以获得显著的性能优势。
Many scientific applications operate on data sets that span hundreds of Gigabytes or even Terabytes in size. Large data sets often use compression to reduce the size of the files. Yet as of today, parallel I/O libraries do not support reading and writing compressed files, necessitating either expensive sequential compression/decompression operations before/after the simulation, or omitting advanced features of parallel I/O libraries, such as collective I/O operations. This paper introduces parallel I/O on compressed data files, discusses the key challenges, requirements, and solutions for supporting compressed data files in MPI I/O, as well as limitations on some MPI I/O operations when using compressed data files. The paper details handling of individual read and write operations of compressed data files, and presents an extension to the two-phase collective I/O algorithm to support data compression. The paper further presents and evaluates an implementation based on the Snappy compression library and the OMPIO parallel I/O framework. The performance evaluation using multiple data sets demonstrate significant performance benefits when using data compression on a parallel BeeGFS file system.