Collaborative Research: Frameworks: FZ: A fine-tunable cyberinfrastructure framework to streamline specialized lossy compression development

数据是科学方法论的第四个支柱。然而，由模拟和仪器设施产生的科学数据的数量和速度的快速增长为许多科学提出了严重的存储容量、存储和网络带宽以及数据分析挑战。这些挑战最终限制了促进繁荣和福利的研究发现。许多研究小组正在探索使用数据缩减技术来解决这些挑战，因为科学数据的有损压缩提供了可靠，高速和高保真的解决方案。然而，现有的通用有损压缩器通常不对应于用户特定的应用、用例以及在缩减、速度和信息保存方面的要求。因此，许多用于科学数据的有损压缩器的潜在用户开发他们自己的专用有损压缩软件，这需要压缩器专家和领域科学家之间的巨大合作，需要广泛的编码来优化多个平台上的性能，并且通常导致冗余的研究和开发工作。该项目旨在创建一个名为FZ的框架，通过提供一个全面的生态系统来彻底改变专用有损压缩器的开发，使科学用户能够直观地研究，编写，实现和测试专用有损压缩器，这些压缩器来自为异构平台优化的预开发，高性能数据减少模块库。该项目还通过提高科学数据管理、压缩和可视化方面与计算有关的课程的质量，并通过在四所大学开展外联活动，为本科生和研究生的教育和培训作出贡献。该项目构建了FZ，一个直观的网络基础设施，用于专门的有损压缩机的组成，通过调整，组合和扩展SZ有损压缩机，LibPressio统一压缩界面，压缩机配置的OptZConfig优化器，Z-checker和QCAT压缩质量分析工具以及Paraview和VTK可视化工具的多种现有功能。该项目有三个重点：（1）开发编程接口和压缩器生成器，以从Python等高级语言创建新的压缩器并优化其执行。(2)它重构了SZ有损压缩器基础设施，以实现各种数据转换模块的细粒度可组合性，并集成了非均匀压缩功能，新的预处理，去相关，近似和熵编码数据转换模块，以产生专门的有损压缩器。(3)它提供了交互式可视化、质量评估和图形用户界面（GUI）工具，这些工具适应并扩展了现有功能，以自动搜索优化的有损压缩模块组合，并识别相关的压缩比、速度、优质贸易-该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的评估被认为值得支持。影响审查标准。

Data is the fourth pillar of the science methodology. However, rapidly expanding volumes and velocities of scientific data generated by simulation and instrument facilities present serious storage capacity, storage and network bandwidth, and data analysis challenges for many sciences. These challenges ultimately limit research discovery which would promote prosperity and welfare. Many research groups are exploring the use of data reduction techniques to address these challenges because lossy compression for scientific data offers a reliable, high-speed, and high-fidelity solution. However, existing generic lossy compressors often do not correspond to user-specific applications, use cases, and requirements in terms of reduction, speed, and information preservation. Hence, many potential users of lossy compressors for scientific data develop their own specialized lossy compression software, an effort that requires tremendous collaboration between compressor experts and domain scientists, demands extensive coding to optimize performance on multiple platforms, and often leads to redundant research and development efforts. This project aims to create a framework, called FZ, that revolutionizes the development of specialized lossy compressors by providing a comprehensive ecosystem to enable scientific users to intuitively research, compose, implement, and test specialized lossy compressors from a library of pre-developed, high-performance data reduction modules optimized for heterogeneous platforms. This project also contributes to the education and training of undergraduate and graduate students by enhancing the quality of computing-related curricula in scientific data management, compression, and visualization and through outreach activities at four universities. This project builds FZ, an intuitive cyberinfrastructure for the composition of specialized lossy compressors, by adapting, combining, and extending multiple existing capabilities from the SZ lossy compressor, the LibPressio unifying compression interface, the OptZConfig optimizer of compressor configurations, the Z-checker and QCAT compression quality analysis tools, and the Paraview and VTK visualization tools. The project has three thrusts: (1) It develops programming interfaces and a compressor generator to create new compressors from high-level languages such as Python and optimize their execution. (2) It refactors the SZ lossy compressors infrastructure to enable fine-grained composability of a large diversity of data transformation modules and integrate non-uniform compression capabilities, new preprocessing, decorrelation, approximation, and entropy coding data transformation modules to produce specialized lossy compressors. (3) It provides interactive visualization, quality assessment, and graphical user interface (GUI) tools that adapt and extend existing capabilities to automatically search optimized lossy compression module compositions and to identify relevant compression ratio, speed, and quality trade-offs for their use cases.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.