Collaborative Research: Elements: ROCCI: Integrated Cyberinfrastructure for In Situ Lossy Compression Optimization Based on Post Hoc Analysis Requirements

今天的模拟和先进的仪器正在产生大量的数据，给科学家带来了很大的存储和I/O负担。错误有界有损压缩器在控制数据失真的同时，又能显著减少数据量，近年来得到了广泛的应用。然而，在实践中仍有很大的差距。一方面，压缩误差对科学研究的影响还没有得到很好的认识，因此如何为有损压缩设定一个合适的误差界是一个非常具有挑战性的问题。另一方面，由于不同压缩技术的优缺点以及应用程序和数据集的不同特性，如何选择最适合的压缩技术并在科学应用代码中自动运行是一个非常重要的问题。该项目旨在开发面向需求的压缩网络基础设施（ROCCI），用于数据密集型领域，如天体物理学和材料科学，它可以根据用户对其事后分析的要求，在运行时自动选择和运行最适合的有损压缩器。该项目的总体目标是提供一系列完整的自动功能和服务，允许用户在科学模拟或数据采集的运行期间透明地运行最适合的压缩机。该项目通过三个关键推力推进知识和理解：(1)它构建了一个有效的层，通过利用现有的压缩适配器库（LibPressio）和压缩评估库（Z-checker），与不同的有损压缩器和对数据保真度的各种事后分析需求进行互操作；(2)开发高效的发动机，根据用户的事后分析要求，通过优化设置来确定最适合的压缩机；(3)开发了一个用户友好的基础设施，通过HDF5动态过滤机制集成压缩优化和执行。该项目特别针对宇宙学和材料科学应用及其在实践中使用有损压缩机的具体要求。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Today’s simulations and advanced instruments are producing vast volumes of data, presenting a major storage and I/O burden for scientists. Error-bounded lossy compressors, which can significantly reduce the data volume while controlling data distortion with a constant error bound, have been developed for years. However, a significant gap still remains in practice. On the one hand, the impact of the compression errors on scientific research is not well understood, so how to set an appropriate error bound for lossy compression is very challenging. On the other hand, how to select the best fit compression technology and run it automatically in scientific application codes is non-trivial because of strengths and weaknesses of different compression techniques and diverse characteristics of applications and datasets. This project aims to develop a Requirement-Oriented Compression Cyber-Infrastructure (ROCCI) for data-intensive domains such as astrophysics and materials science, which can select and run the best fit lossy compressor automatically at runtime, in terms of user's requirement on their post hoc analysis.The overarching goal of this project is to offer a complete series of automatic functions and services allowing users to transparently run the best fit compressor at runtime during the scientific simulations or data acquisition. This project advances knowledge and understanding with three key thrusts: (1) it builds an efficient layer to interoperate with different lossy compressors and diverse post hoc analysis requirements on data fidelity by leveraging an existing compression adaptor library (LibPressio) and compression assessment library (Z-checker); (2) it develops an efficient engine to determine the best fit compressor with optimized settings based on user’s post-hoc analysis requirements; and (3) it develops a user-friendly infrastructure that integrates compression optimization and execution via the HDF5 dynamic filter mechanism. This project particularly targets cosmology and materials science applications and their specific requirements of using lossy compressors in practice.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.