Collaborative Research: Elements: ROCCI: Integrated Cyberinfrastructure for In Situ Lossy Compression Optimization Based on Post Hoc Analysis Requirements

今天的模拟和先进仪器产生了大量的数据，给科学家带来了巨大的存储和I/O负担。误差有界有损压缩器是近年来发展起来的一种压缩器，它可以在保持恒定误差界的情况下，有效地减少数据量，同时控制数据失真。然而，在实践中仍然存在很大差距。一方面，压缩误差对科学研究的影响还没有得到很好的理解，因此如何为有损压缩设置合适的误差范围是非常具有挑战性的。另一方面，如何选择最适合的压缩技术并在科学应用代码中自动运行它是不平凡的，因为不同压缩技术的优点和缺点以及应用程序和数据集的不同特征。该项目旨在为天体物理学和材料科学等数据密集型领域开发一个面向有损压缩的网络基础设施（ROCCI），它可以在运行时自动选择和运行最适合的有损压缩器，该项目的总体目标是提供一系列完整的自动化功能和服务，允许用户透明地运行最适合的在科学模拟或数据采集过程中，压缩机在运行时运行。该项目通过以下三个关键点推进知识和理解：（1）通过利用现有的压缩适配器库，构建了一个有效的层，以与不同的有损压缩器和对数据保真度的各种事后分析需求进行互操作（LibPressio）和压缩评估库（2）它开发了一个有效的引擎，以基于用户的事后分析要求确定具有优化设置的最佳匹配压缩机;以及（3）它开发了一个用户友好的基础设施，该基础设施通过HDF 5动态过滤器机制集成了压缩优化和执行。该项目特别针对宇宙学和材料科学应用及其在实践中使用有损压缩机的具体要求。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Today’s simulations and advanced instruments are producing vast volumes of data, presenting a major storage and I/O burden for scientists. Error-bounded lossy compressors, which can significantly reduce the data volume while controlling data distortion with a constant error bound, have been developed for years. However, a significant gap still remains in practice. On the one hand, the impact of the compression errors on scientific research is not well understood, so how to set an appropriate error bound for lossy compression is very challenging. On the other hand, how to select the best fit compression technology and run it automatically in scientific application codes is non-trivial because of strengths and weaknesses of different compression techniques and diverse characteristics of applications and datasets. This project aims to develop a Requirement-Oriented Compression Cyber-Infrastructure (ROCCI) for data-intensive domains such as astrophysics and materials science, which can select and run the best fit lossy compressor automatically at runtime, in terms of user's requirement on their post hoc analysis.The overarching goal of this project is to offer a complete series of automatic functions and services allowing users to transparently run the best fit compressor at runtime during the scientific simulations or data acquisition. This project advances knowledge and understanding with three key thrusts: (1) it builds an efficient layer to interoperate with different lossy compressors and diverse post hoc analysis requirements on data fidelity by leveraging an existing compression adaptor library (LibPressio) and compression assessment library (Z-checker); (2) it develops an efficient engine to determine the best fit compressor with optimized settings based on user’s post-hoc analysis requirements; and (3) it develops a user-friendly infrastructure that integrates compression optimization and execution via the HDF5 dynamic filter mechanism. This project particularly targets cosmology and materials science applications and their specific requirements of using lossy compressors in practice.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.