Optimizing Lossy Compression with Adjacent Snapshots for N-body Simulation Data
Optimizing Lossy Compression with Adjacent Snapshots for N-body Simulation Data
复制标题
使用相邻快照优化 N 体仿真数据的有损压缩
DOI：
10.1109/bigdata.2018.8622101
复制
发表时间：
2018
期刊：
影响因子：
--
通讯作者：
F. Cappello
中科院分区：
文献类型：
--
作者：
Sihuan Li;S. Di;Xin Liang;Zizhong Chen;F. Cappello
今天的N体模拟产生了非常大量的数据。例如，硬件/混合加速宇宙学代码（HACC）可以模拟数万亿个粒子，产生数十PB的数据存储在并行文件系统中。在本文中，我们设计并实现了一个有效的，在原位误差有界的有损压缩器显着减少N体模拟的数据大小。我们的压缩器不仅可以为N体模拟研究人员节省大量的存储空间，而且还可以在有限的内存和计算开销的情况下大大提高I/O性能。我们的贡献是三方面的。(1)我们提出了一个有效的数据压缩模型，利用宇宙学数据在空间和时间维度的连续性以及不同领域之间的物理相关性。(2)我们提出了一个轻量级的，高效的对齐机制，在模拟中，这是整个压缩过程中的一个基本步骤，在相邻的快照对齐无序粒子。我们还通过探索最佳拟合数据预测策略和优化基于空间的压缩与基于时间的压缩的频率来优化压缩质量。(3)我们评估我们的压缩机使用宇宙学模拟包和分子动力学模拟数据-两个主要类别中的N体模拟域。实验表明，在相同的数据失真下，我们的解决方案产生高达43%的速度场和高达300%的位置场比其他国家的最先进的压缩器（包括SZ，ZFP，NUMARCK，和抽取）。使用我们的压缩器，与第二好的压缩器相比，HACC数据的总体I/O时间减少了20%。
Today’s N-body simulations are producing extremely large amounts of data. The Hardware/Hybrid Accelerated Cosmology Code (HACC), for example, may simulate trillions of particles, producing tens of petabytes of data to store in a parallel file system, according to the HACC users. In this paper, we design and implement an efficient, in situ error-bounded lossy compressor to significantly reduce the data size for N-body simulations. Not only can our compressor save significant storage space for N-body simulation researchers, but it can also improve the I/O performance considerably with limited memory and computation overhead. Our contribution is threefold. (1) We propose an efficient data compression model by leveraging the consecutiveness of the cosmological data in both space and time dimensions as well as the physical correlation across different fields. (2) We propose a lightweight, efficient alignment mechanism to align the disordered particles across adjacent snapshots in the simulation, which is a fundamental step in the whole compression procedure. We also optimize the compression quality by exploring best-fit data prediction strategies and optimizing the frequencies of the space-based compression vs. time-based compression. (3) We evaluate our compressor using both a cosmological simulation package and molecular dynamics simulation data—two major categories in the N-body simulation domain. Experiments show that under the same distortion of data, our solution produces up to 43% higher compression ratios on the velocity field and up to 300% higher on the position field than do other state-of-the-art compressors (including SZ, ZFP, NUMARCK, and decimation). With our compressor, the overall I/O time on HACC data is reduced by up to 20% compared with the second-best compressor.