Elements: Advanced Lossless and Lossy Compression Algorithms for netCDF Datasets in Earth and Engineering Sciences (CANDEE)

数据压缩用于通过减少存储空间和下载时间来更有效地存储和传输数字数据，如音乐、电视和卫星测量。该项目提供的压缩软件代理将促进现代压缩技术在许多科学分支中的采用。压缩器有两种类型：无损压缩器，完全保留原始信息的压缩器;和有损压缩器，不可挽回地丢弃部分“信号”以进一步改进压缩的压缩器。现代无损和有损压缩在效率，速度和保真度方面的改进是惊人的，并将使研究人员能够模拟，存储和分析恒星演化，化学反应和飓风形成等现象，而无需额外的存储成本。由于数字存储会消耗电力，因此更好的压缩也会减少电力消耗和相关的温室气体排放。该项目将开发必要的软件基础设施，使科学研究人员能够无缝地转换其应用程序，以产生和使用最先进的无损技术存储的数据，并通过比任何其他技术更准确的新有损技术存储数据。两种最广泛使用的自描述数据集存储格式，HDF 5和netCDF 4，默认情况下只支持一种专利无障碍无损压缩格式，在20世纪90年代标准化的古老的DEFLATE算法。我们的项目将开发一个动态的和可扩展的现代压缩机和解压缩机（编解码器）的科学数据的软件库，称为社区编解码器库（CCR）。我们将使用尖端的开源压缩技术来填充CCR，包括LZ 4、Facebook的Zstandard和Google的Snappy编解码器，并将为CCR实现默认的netCDF支持。顺序的先有损后无损压缩提高了压缩/解压缩的大小和速度，但目前执行起来很繁琐。我们将在我们广泛使用的netCDF Operators软件包中实现一种用户友好的方法，将编解码器“链接”到内存中的顺序操作中（不需要中间文件）。我们还将产生一个新的精度保持有损编解码器，粒度位疏导，具有无与伦比的压缩比和统计精度。技术成功将通过压缩一个典型的地球科学/工程“大数据”项目的规模和速度改进来评估，耦合模型相互比较项目第6版（CMIP 6）。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

Data compression is used to store and transmit digital data such as music, television, and satellite measurements more efficiently by reducing storage space and download times. The compression software broker that this project provides will facilitate the adoption of modern compression techniques in many branches of science. Compressors come in two flavors: lossless, those that perfectly preserve the original information; and lossy, those that irretrievably discard parts of the "signal" to further improve compression. Modern lossless and lossy compression improvements in efficiency, speed, and fidelity, are striking and will benefit critical research areas by permitting researchers to simulate, store, and analyze phenomena such as stellar evolution, chemical reactions, and hurricane formation at finer detail than before, with no extra storage costs. Since digital storage consumes power, better compression also reduces power consumption and associated greenhouse gas emissions. This project will develop the software infrastructure necessary for scientific researchers to seamlessly shift their applications to produce and use data stored with state-of-the-art lossless techniques, and by new lossy techniques that are more accurate than any others.The two most widely-used self-describing dataset storage formats, HDF5 and netCDF4, support by default only one patent unencumbered lossless compression format, the venerable DEFLATE algorithm standardized in the 1990s. Our project will develop a dynamic and extensible software library of modern COmpressors and DECompressors (codecs) for scientific data called the Community Codec Repository (CCR). We will populate the CCR with cutting-edge open-source compression technology, including the LZ4, Facebook's Zstandard, and Google's Snappy codecs, and will implement default netCDF support for the CCR. Sequential lossy-then-lossless compression improves both the size and speed of compression/decompression yet is currently tedious to perform. We will implement a user-friendly method to "chain" codecs into sequential operations in memory (no intermediate files required) in our widely used netCDF Operators software package. We will also produce a new precision-preserving lossy codec, Granular Bit Grooming, that has unsurpassed compression ratio and statistical accuracy. Technical success will be evaluated by the size and speed improvements of compressing a prototypical geoscience/engineering "big data" project, the Coupled Model Intercomparison Project version 6 (CMIP6).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.