CDS&E: Collaborative Research: HyLoC: Objective-driven Adaptive Hybrid Lossy Compression Framework for Extreme-Scale Scientific Applications

今天的极端规模的科学模拟和仪器正在产生大量无法有效传输或存储的数据。有损压缩是一种会导致数据失真的数据压缩方法，它可以在保持数据保真度的同时显著减小数据量，因此被认为是一种很有前途的解决方案。然而，现有的有损压缩方法可能并不总是有效地工作在特定应用中使用的所有数据集，因为它们的独特和多样的特性。此外，压缩质量和性能方面的用户目标可能会因应用程序、数据集或环境而异。该项目旨在开发一个混合有损压缩框架，以自动构建最适合数据密集型科学研究中不同用户目标的压缩。提供教育和参与活动，以开发与科学数据压缩相关的新课程，并促进与国家实验室的研究合作。设计一个高效，自适应，混合框架，始终可以选择最适合的压缩策略是不平凡的，因为现有的最先进的有损压缩方法是根据不同的原则开发的。该项目有三个阶段的研究计划。首先，该项目将最先进的误差有界有损压缩方法分为多个阶段，并有效地模拟了工作效率（例如，压缩比、误差、速度）。其次，该项目开发了一个松散耦合的框架，将解耦的压缩阶段聚合在一起，并探索尽可能多的由不同阶段组成的压缩管道，以优化经典的压缩效率，包括压缩质量和性能。第三，该项目优化了与外部设备和资源相关的合成数据移动性能，例如I/O性能。该团队评估了多个极端规模科学应用的拟议框架，包括宇宙学模拟，光源仪器数据分析，量子电路模拟和气候模拟。该项目可能会创造出能够提高存储可用性并改善极端规模科学应用性能的技术，为新发现创造机会。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Today's extreme-scale scientific simulations and instruments are producing huge amounts of data that cannot be transmitted or stored effectively. Lossy compression, a data compression approach leading to certain data distortion, has been considered as a promising solution, because it can significantly reduce the data size while maintaining high data fidelity. However, the existing lossy compression methods may not always work effectively on all datasets used in specific applications because of their distinct and diverse characteristics. Moreover, the user objectives in compression quality and performance may vary with applications, datasets or circumstances. This project aims to develop a hybrid lossy compression framework to automatically construct the best-fit compression for diverse user objectives in data-intensive scientific research. Educational and engagement activities are provided to develop new curriculum related to scientific data compression and promote research collaborations with national laboratories.Designing an efficient, adaptive, hybrid framework that can always choose the best-fit compression strategy is nontrivial, since existing state-of-the-art lossy compression methods are developed with distinct principles. The project has a three-stage research plan. First, the project decouples the state-of-the-art error-bounded lossy compression approaches into multiple stages and effectively models the working efficiency (e.g., compression ratio, error, speed) of particular approaches in each stage. Second, the project develops a loosely-coupled framework to aggregate the decoupled compression stages together and also explores as many compression pipelines composed of different stages as possible, to optimize the classic compression efficiency, including compression quality and performance. Third, the project optimizes the synthetic data-movement performance regarding the external devices and resources, such as I/O performance. The team evaluates the proposed framework on multiple extreme-scale scientific applications, including cosmological simulations, light source instrument data analytics, quantum circuit simulations, and climate simulations. The project may create technologies that can increase the storage availability and improve the performance for extreme-scale scientific applications, opening opportunities for new discoveries.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.