Efficient and Flexible Hierarchical Data Layouts for a Unified Encoding of Scalar Field Precision and Resolution
Efficient and Flexible Hierarchical Data Layouts for a Unified Encoding of Scalar Field Precision and Resolution
复制标题
DOI：
10.1109/tvcg.2020.3030381
复制
发表时间：
2020-10
影响因子：
5.2
通讯作者：
D. Hoang;B. Summa;H. Bhatia;Peter Lindstrom;Pavol Klacansky;W. Usher;P. Bremer;Valerio Pascucci
中科院分区：
文献类型：
--
作者：
D. Hoang;B. Summa;H. Bhatia;Peter Lindstrom;Pavol Klacansky;W. Usher;P. Bremer;Valerio Pascucci
为了解决不断增长的科学数据的大小，使数据移动的一个主要障碍分析的问题，我们介绍了一种新的标量场编码：一个统一的树的分辨率和精度，专门构造，使有效的削减对应于合理的近似原始字段的精度-分辨率空间。此外，我们引入了一个高度灵活的编码，这样的树，形成一个参数化的家庭的数据层次结构。我们讨论了不同的参数选择如何在实践中导致不同的权衡，并展示了特定的选择如何导致已知的数据表示方案，如zfp [52]，idx [58]和jpeg 2000 [76]。最后，我们提供了系统级的细节和经验证据，这样的层次结构如何促进常见的近似查询，最小的数据移动和时间，使用现实世界的数据集，从几千兆字节到近一个TB的大小。实验表明，我们的新策略相结合的分辨率和精度的减少是有竞争力的国家的最先进的压缩技术方面的数据质量，同时显着更灵活和数量级更快，并需要显着减少资源。
To address the problem of ever-growing scientific data sizes making data movement a major hindrance to analysis, we introduce a novel encoding for scalar fields: a unified tree of resolution and precision, specifically constructed so that valid cuts correspond to sensible approximations of the original field in the precision-resolution space. Furthermore, we introduce a highly flexible encoding of such trees that forms a parameterized family of data hierarchies. We discuss how different parameter choices lead to different trade-offs in practice, and show how specific choices result in known data representation schemes such as zfp [52], idx [58], and jpeg2000 [76]. Finally, we provide system-level details and empirical evidence on how such hierarchies facilitate common approximate queries with minimal data movement and time, using real-world data sets ranging from a few gigabytes to nearly a terabyte in size. Experiments suggest that our new strategy of combining reductions in resolution and precision is competitive with state-of-the-art compression techniques with respect to data quality, while being significantly more flexible and orders of magnitude faster, and requiring significantly reduced resources.