CDS&E: Collaborative Research: HyLoC: Objective-driven Adaptive Hybrid Lossy Compression Framework for Extreme-Scale Scientific Applications

今天的极端规模的科学模拟和仪器正在产生大量的数据，不能有效地传输或存储。有损压缩是一种导致一定数据失真的数据压缩方法，它被认为是一种很有前途的解决方案，因为它可以在保持高数据保真度的同时显着减小数据大小。然而，现有的有损压缩方法可能并不总是对特定应用中使用的所有数据集有效，因为它们具有不同的特性。此外，用户在压缩质量和性能方面的目标可能因应用程序、数据集或环境而异。本项目旨在开发一种混合有损压缩框架，以自动构建最适合数据密集型科学研究中不同用户目标的压缩。提供教育和参与活动，以制定与科学数据压缩有关的新课程，并促进与国家实验室的研究合作。设计一个高效的、自适应的混合框架，总是能够选择最适合的压缩策略，这是非常重要的，因为现有的最先进的有损压缩方法是根据不同的原则开发的。该项目有三个阶段的研究计划。首先，该项目将最先进的误差有界有损压缩方法解耦到多个阶段，并在每个阶段有效地模拟特定方法的工作效率（例如压缩比、误差、速度）。其次，该项目开发了一个松耦合框架，将解耦的压缩阶段聚合在一起，并探索尽可能多的由不同阶段组成的压缩管道，以优化经典压缩效率，包括压缩质量和性能。第三，针对外部设备和资源优化综合数据移动性能，如I/O性能。该团队在多个极端尺度的科学应用中评估了所提出的框架，包括宇宙学模拟、光源仪器数据分析、量子电路模拟和气候模拟。该项目可能会创造出能够提高存储可用性和提高极端规模科学应用性能的技术，为新发现创造机会。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Today's extreme-scale scientific simulations and instruments are producing huge amounts of data that cannot be transmitted or stored effectively. Lossy compression, a data compression approach leading to certain data distortion, has been considered as a promising solution, because it can significantly reduce the data size while maintaining high data fidelity. However, the existing lossy compression methods may not always work effectively on all datasets used in specific applications because of their distinct and diverse characteristics. Moreover, the user objectives in compression quality and performance may vary with applications, datasets or circumstances. This project aims to develop a hybrid lossy compression framework to automatically construct the best-fit compression for diverse user objectives in data-intensive scientific research. Educational and engagement activities are provided to develop new curriculum related to scientific data compression and promote research collaborations with national laboratories.Designing an efficient, adaptive, hybrid framework that can always choose the best-fit compression strategy is nontrivial, since existing state-of-the-art lossy compression methods are developed with distinct principles. The project has a three-stage research plan. First, the project decouples the state-of-the-art error-bounded lossy compression approaches into multiple stages and effectively models the working efficiency (e.g., compression ratio, error, speed) of particular approaches in each stage. Second, the project develops a loosely-coupled framework to aggregate the decoupled compression stages together and also explores as many compression pipelines composed of different stages as possible, to optimize the classic compression efficiency, including compression quality and performance. Third, the project optimizes the synthetic data-movement performance regarding the external devices and resources, such as I/O performance. The team evaluates the proposed framework on multiple extreme-scale scientific applications, including cosmological simulations, light source instrument data analytics, quantum circuit simulations, and climate simulations. The project may create technologies that can increase the storage availability and improve the performance for extreme-scale scientific applications, opening opportunities for new discoveries.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.