Elements: Advanced Lossless and Lossy Compression Algorithms for netCDF Datasets in Earth and Engineering Sciences (CANDEE)

数据压缩通过减少存储空间和下载时间，用于更高效地存储和传输音乐、电视和卫星测量等数字数据。该项目提供的压缩软件代理将促进现代压缩技术在许多科学分支中的采用。压缩程序有两种类型：无损压缩程序，即完美地保存原始信息的压缩程序；有损压缩程序，即不可挽回地丢弃部分“信号”以进一步提高压缩效率的压缩程序。现代无损压缩在效率、速度和保真度方面的改进是引人注目的，并将使关键研究领域受益，因为它允许研究人员以比以前更精细的细节模拟、存储和分析恒星演化、化学反应和飓风形成等现象，而不需要额外的存储成本。由于数字存储会消耗电力，因此更好的压缩也会减少电力消耗和相关的温室气体排放。该项目将为科学研究人员开发必要的软件基础设施，以无缝地转移他们的应用程序，以生产和使用以最先进的无损技术和比其他任何技术更准确的新的有损技术存储的数据。两种最广泛使用的自我描述数据集存储格式HDF5和netCDF4默认只支持一种专利无阻碍无损压缩格式，即在20世纪90年代标准化的历史悠久的通货紧缩算法。我们的项目将为科学数据开发一个动态和可扩展的现代压缩器和解压缩器(编解码器)软件库，称为社区编解码库(CCR)。我们将用尖端的开源压缩技术填充CCR，包括LZ4、Facebook的Z标准和Google的Snappy编解码器，并将为CCR实现默认的netCDF支持。顺序有损然后无损压缩提高了压缩/解压缩的大小和速度，但目前执行起来很繁琐。我们将在我们广泛使用的netCDF操作员软件包中实现一种用户友好的方法，将编解码链到内存中的顺序操作中(不需要中间文件)。我们还将推出一种新的保持精度的有损编解码器-颗粒比特疏导，具有无与伦比的压缩比和统计精度。技术成功将通过压缩一个典型的地球科学/工程“大数据”项目--耦合模型比较项目版本6(CMIP6)--的规模和速度改进来评估。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Data compression is used to store and transmit digital data such as music, television, and satellite measurements more efficiently by reducing storage space and download times. The compression software broker that this project provides will facilitate the adoption of modern compression techniques in many branches of science. Compressors come in two flavors: lossless, those that perfectly preserve the original information; and lossy, those that irretrievably discard parts of the "signal" to further improve compression. Modern lossless and lossy compression improvements in efficiency, speed, and fidelity, are striking and will benefit critical research areas by permitting researchers to simulate, store, and analyze phenomena such as stellar evolution, chemical reactions, and hurricane formation at finer detail than before, with no extra storage costs. Since digital storage consumes power, better compression also reduces power consumption and associated greenhouse gas emissions. This project will develop the software infrastructure necessary for scientific researchers to seamlessly shift their applications to produce and use data stored with state-of-the-art lossless techniques, and by new lossy techniques that are more accurate than any others.The two most widely-used self-describing dataset storage formats, HDF5 and netCDF4, support by default only one patent unencumbered lossless compression format, the venerable DEFLATE algorithm standardized in the 1990s. Our project will develop a dynamic and extensible software library of modern COmpressors and DECompressors (codecs) for scientific data called the Community Codec Repository (CCR). We will populate the CCR with cutting-edge open-source compression technology, including the LZ4, Facebook's Zstandard, and Google's Snappy codecs, and will implement default netCDF support for the CCR. Sequential lossy-then-lossless compression improves both the size and speed of compression/decompression yet is currently tedious to perform. We will implement a user-friendly method to "chain" codecs into sequential operations in memory (no intermediate files required) in our widely used netCDF Operators software package. We will also produce a new precision-preserving lossy codec, Granular Bit Grooming, that has unsurpassed compression ratio and statistical accuracy. Technical success will be evaluated by the size and speed improvements of compressing a prototypical geoscience/engineering "big data" project, the Coupled Model Intercomparison Project version 6 (CMIP6).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.