High-Quality and Low-Memory-Footprint Progressive Decoding of Large-Scale Particle Data
High-Quality and Low-Memory-Footprint Progressive Decoding of Large-Scale Particle Data
复制标题
DOI：
10.1109/ldav53230.2021.00011
复制
发表时间：
2021-10
期刊：
影响因子：
--
通讯作者：
D. Hoang;H. Bhatia;P. Lindstrom;Valerio Pascucci
中科院分区：
文献类型：
--
作者：
D. Hoang;H. Bhatia;P. Lindstrom;Valerio Pascucci
粒子表示法常用于大规模模拟和观测中，经常会创建包含数百万个甚至更多粒子的数据集。由于其规模庞大，这类数据集难以高效地存储、传输和分析。数据压缩是一种有前景的解决方案；然而，缺乏有效的粒子数据压缩方法，也没有被社区认可的标准技术。当前的技术要么能很好地压缩小数据，但应用于大数据时需要大量计算资源，要么能处理大数据但不注重压缩，导致每存储一位的重建质量较低。在本文中，我们提出了针对基于树的粒子压缩方法的创新，改善了在高质量和低内存占用之间的权衡，以用于大型粒子数据集的压缩和解压缩。受懒小波变换的启发，我们引入了一种划分空间的新方法，它允许以低成本的深度优先遍历粒子层次结构来广泛覆盖空间。我们还设计了新的数据自适应遍历顺序，与传统的与数据无关的顺序（如广度优先和深度优先遍历）相比，显著降低了重建误差。新的划分和遍历方案被用于构建新的粒子层次结构，在产生低重建误差的同时，可以用渐近恒定的内存占用进行遍历。我们对大型粒子数据进行编码和（有损）解码的解决方案是一种灵活的基于块的层次结构，它支持渐进式、随机访问和误差驱动解码，其中误差启发式方法可由用户提供。最后，通过大量实验，我们证明了所提出的技术在组合使用以及与现有方法在各种科学粒子数据集上独立使用时的有效性和灵活性。
Particle representations are used often in large-scale simulations and observations, frequently creating datasets containing several millions of particles or more. Due to their sheer size, such datasets are difficult to store, transfer, and analyze efficiently. Data compression is a promising solution; however, effective approaches to compress particle data are lacking and no community-standard and accepted techniques exist. Current techniques are designed either to compress small data very well but require high computational resources when applied to large data, or to work with large data but without a focus on compression, resulting in low reconstruction quality per bit stored. In this paper, we present innovations targeting tree-based particle compression approaches that improve the tradeoff between high quality and low memory-footprint for compression and decompression of large particle datasets. Inspired by the lazy wavelet transform, we introduce a new way of partitioning space, which allows a low-cost depth-first traversal of a particle hierarchy to cover the space broadly. We also devise novel data-adaptive traversal orders that significantly reduce reconstruction error compared to traditional data-agnostic orders such as breadth-first and depth-first traversals. The new partitioning and traversal schemes are used to build novel particle hierarchies that can be traversed with asymptotically constant memory footprint while incurring low reconstruction error. Our solution to encoding and (lossy) decoding of large particle data is a flexible block-based hierarchy that supports progressive, random-access, and error-driven decoding, where error heuristics can be supplied by the user. Finally, through extensive experimentation, we demonstrate the efficacy and the flexibility of the proposed techniques when combined as well as when used independently with existing approaches on a wide range of scientific particle datasets.