Collaborative Research: Frameworks: FZ: A fine-tunable cyberinfrastructure framework to streamline specialized lossy compression development

数据是科学方法论的第四大支柱。然而，模拟和仪器设备产生的科学数据的数量和速度迅速增长，给许多科学带来了严重的存储容量、存储和网络带宽以及数据分析挑战。这些挑战最终限制了将促进繁荣和福利的研究发现。许多研究小组正在探索使用数据缩减技术来应对这些挑战，因为科学数据的有损压缩提供了可靠、高速和高保真的解决方案。然而，现有的通用有损压缩器在减少、速度和信息保存方面往往不符合用户特定的应用、用例和要求。因此，许多科学数据有损压缩器的潜在用户开发他们自己的专门有损压缩软件，这一努力需要压缩机专家和领域科学家之间的巨大合作，需要广泛的编码来优化在多个平台上的性能，并且经常导致重复的研究和开发工作。该项目旨在创建一个名为FZ的框架，通过提供一个全面的生态系统，使科学用户能够直观地研究、合成、实施和测试针对不同平台优化的预先开发的高性能数据压缩模块库中的专用有损压缩程序，从而彻底改变专用有损压缩程序的开发。该项目还通过在四所大学开展外联活动，提高与计算机有关的科学数据管理、压缩和可视化课程的质量，为本科生和研究生的教育和培训作出贡献。该项目通过调整、组合和扩展SZ有损压缩机、LibPressio统一压缩接口、压缩机配置的OptZConfig优化器、Z-Checker和QCAT压缩质量分析工具以及Paraview和VTK可视化工具的多种现有功能，构建了FZ，这是一个直观的网络基础设施，用于合成专门的有损压缩机。该项目有三个推动力：(1)开发编程接口和压缩机生成器，以从高级语言(如Python语言)创建新的压缩机，并优化其执行。(2)对SZ有损压缩器基础设施进行重构，以实现多种数据转换模块的细粒度可组合性，并集成非均匀压缩能力、新的预处理、去相关、近似和熵编码数据转换模块，以产生专门的有损压缩器。(3)它提供交互式可视化、质量评估和图形用户界面(GUI)工具，以调整和扩展现有功能，以自动搜索优化的有损压缩模块组合，并确定其使用案例的相关压缩比、速度和质量权衡。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Data is the fourth pillar of the science methodology. However, rapidly expanding volumes and velocities of scientific data generated by simulation and instrument facilities present serious storage capacity, storage and network bandwidth, and data analysis challenges for many sciences. These challenges ultimately limit research discovery which would promote prosperity and welfare. Many research groups are exploring the use of data reduction techniques to address these challenges because lossy compression for scientific data offers a reliable, high-speed, and high-fidelity solution. However, existing generic lossy compressors often do not correspond to user-specific applications, use cases, and requirements in terms of reduction, speed, and information preservation. Hence, many potential users of lossy compressors for scientific data develop their own specialized lossy compression software, an effort that requires tremendous collaboration between compressor experts and domain scientists, demands extensive coding to optimize performance on multiple platforms, and often leads to redundant research and development efforts. This project aims to create a framework, called FZ, that revolutionizes the development of specialized lossy compressors by providing a comprehensive ecosystem to enable scientific users to intuitively research, compose, implement, and test specialized lossy compressors from a library of pre-developed, high-performance data reduction modules optimized for heterogeneous platforms. This project also contributes to the education and training of undergraduate and graduate students by enhancing the quality of computing-related curricula in scientific data management, compression, and visualization and through outreach activities at four universities. This project builds FZ, an intuitive cyberinfrastructure for the composition of specialized lossy compressors, by adapting, combining, and extending multiple existing capabilities from the SZ lossy compressor, the LibPressio unifying compression interface, the OptZConfig optimizer of compressor configurations, the Z-checker and QCAT compression quality analysis tools, and the Paraview and VTK visualization tools. The project has three thrusts: (1) It develops programming interfaces and a compressor generator to create new compressors from high-level languages such as Python and optimize their execution. (2) It refactors the SZ lossy compressors infrastructure to enable fine-grained composability of a large diversity of data transformation modules and integrate non-uniform compression capabilities, new preprocessing, decorrelation, approximation, and entropy coding data transformation modules to produce specialized lossy compressors. (3) It provides interactive visualization, quality assessment, and graphical user interface (GUI) tools that adapt and extend existing capabilities to automatically search optimized lossy compression module compositions and to identify relevant compression ratio, speed, and quality trade-offs for their use cases.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.