Collaborative Research: Frameworks: FZ: A fine-tunable cyberinfrastructure framework to streamline specialized lossy compression development

数据是科学方法论的第四大支柱。然而，由模拟和仪器设备产生的科学数据的数量和速度迅速扩大，给许多科学带来了严重的存储容量、存储和网络带宽以及数据分析挑战。这些挑战最终限制了能够促进繁荣和福利的研究发现。许多研究小组正在探索使用数据约简技术来解决这些挑战，因为有损压缩为科学数据提供了可靠、高速和高保真的解决方案。然而，现有的通用有损压缩器在减少、速度和信息保存方面通常不符合用户特定的应用程序、用例和需求。因此，许多用于科学数据的有损压缩器的潜在用户开发了他们自己专门的有损压缩软件，这需要压缩器专家和领域科学家之间的大量合作，需要大量的编码来优化多个平台的性能，并且经常导致冗余的研究和开发工作。该项目旨在创建一个名为FZ的框架，通过提供一个全面的生态系统，使科学用户能够直观地研究、编写、实现和测试专用有损压缩器，从而彻底改变专用有损压缩器的开发，这些专用有损压缩器来自预先开发的高性能数据缩减模块库，针对异构平台进行了优化。本计划亦透过在四所大学举办外展活动，提高有关科学数据管理、压缩及可视化的计算相关课程的质素，以促进本科生及研究生的教育及训练。该项目通过调整、组合和扩展SZ有损压缩器、LibPressio统一压缩接口、OptZConfig压缩配置优化器、Z-checker和QCAT压缩质量分析工具以及Paraview和VTK可视化工具的多种现有功能，构建了FZ，这是一个用于专业有损压缩器组成的直观网络基础设施。该项目有三个重点：(1)开发编程接口和压缩器生成器，以使用Python等高级语言创建新的压缩器并优化其执行。(2)重构了SZ有损压缩器基础设施，实现了大量数据转换模块的细粒度可组合性，并集成了非均匀压缩能力、新的预处理、去相关、近似和熵编码数据转换模块，以产生专门的有损压缩器。(3)它提供了交互式可视化、质量评估和图形用户界面（GUI）工具，这些工具适应并扩展了现有的功能，以自动搜索优化的有损压缩模块组合，并为它们的用例识别相关的压缩比、速度和质量权衡。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Data is the fourth pillar of the science methodology. However, rapidly expanding volumes and velocities of scientific data generated by simulation and instrument facilities present serious storage capacity, storage and network bandwidth, and data analysis challenges for many sciences. These challenges ultimately limit research discovery which would promote prosperity and welfare. Many research groups are exploring the use of data reduction techniques to address these challenges because lossy compression for scientific data offers a reliable, high-speed, and high-fidelity solution. However, existing generic lossy compressors often do not correspond to user-specific applications, use cases, and requirements in terms of reduction, speed, and information preservation. Hence, many potential users of lossy compressors for scientific data develop their own specialized lossy compression software, an effort that requires tremendous collaboration between compressor experts and domain scientists, demands extensive coding to optimize performance on multiple platforms, and often leads to redundant research and development efforts. This project aims to create a framework, called FZ, that revolutionizes the development of specialized lossy compressors by providing a comprehensive ecosystem to enable scientific users to intuitively research, compose, implement, and test specialized lossy compressors from a library of pre-developed, high-performance data reduction modules optimized for heterogeneous platforms. This project also contributes to the education and training of undergraduate and graduate students by enhancing the quality of computing-related curricula in scientific data management, compression, and visualization and through outreach activities at four universities. This project builds FZ, an intuitive cyberinfrastructure for the composition of specialized lossy compressors, by adapting, combining, and extending multiple existing capabilities from the SZ lossy compressor, the LibPressio unifying compression interface, the OptZConfig optimizer of compressor configurations, the Z-checker and QCAT compression quality analysis tools, and the Paraview and VTK visualization tools. The project has three thrusts: (1) It develops programming interfaces and a compressor generator to create new compressors from high-level languages such as Python and optimize their execution. (2) It refactors the SZ lossy compressors infrastructure to enable fine-grained composability of a large diversity of data transformation modules and integrate non-uniform compression capabilities, new preprocessing, decorrelation, approximation, and entropy coding data transformation modules to produce specialized lossy compressors. (3) It provides interactive visualization, quality assessment, and graphical user interface (GUI) tools that adapt and extend existing capabilities to automatically search optimized lossy compression module compositions and to identify relevant compression ratio, speed, and quality trade-offs for their use cases.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.