Evaluating lossy data compression on climate simulation data within a large ensemble
Evaluating lossy data compression on climate simulation data within a large ensemble
复制标题
DOI：
10.5194/gmd-9-4381-2016
复制
发表时间：
2016-12-07
影响因子：
5.1
通讯作者：
Lindstrom, Peter
中科院分区：
文献类型：
--
作者：
Baker, Allison H.;Hammerling, Dorit M.;Lindstrom, Peter
高分辨率地球系统模型模拟会产生大量数据，而保留这些模拟中的数据通常会导致机构存储资源紧张。此外，这些极大的存储需求会对科学目标产生负面影响，例如，通过强制减少数据输出频率、模拟长度或集合大小。为了减少社区地球系统模型（CESM）的数据量，我们提倡使用有损数据压缩技术。虽然有损数据压缩不能完全保留原始数据（如无损压缩那样），但有损技术在存储需求较小方面具有优势。为了保持科学模拟数据的完整性，有损数据压缩对原始数据的影响至少在统计上不应与气候系统的自然变率区分开来，之前对CESM数据的初步研究表明这一目标是可以实现的。然而，为了最终说服气候科学家使用有损数据压缩是可以接受的，我们为气候科学家提供了经过有损数据压缩的公开气候数据的访问权限。我们特别报告了有损数据压缩实验的结果，该实验的输出来自 CESM 大型集合体 (CESM-LE) 社区项目，其中我们要求气候科学家检查与其兴趣相关的数据特征，并尝试确定哪些集合成员已被压缩和重建。我们发现，虽然检测区分特征当然是可能的，但这些特征中明显的压缩效果通常不重要或在后处理分析中消失。此外，我们还进行了多项分析，直接将原始数据与重建数据进行比较，以调查对气候科学至关重要的特定特征的保存或缺乏。总的来说，我们的结论是，将有损数据压缩应用于气候模拟数据既在数据减少方面有利，又在对科学结果的影响方面普遍可接受。
High-resolution Earth system model simulations generate enormous data volumes, and retaining the data from these simulations often strains institutional storage resources. Further, these exceedingly large storage requirements negatively impact science objectives, for example, by forcing reductions in data output frequency, simulation length, or ensemble size. To lessen data volumes from the Community Earth System Model (CESM), we advocate the use of lossy data compression techniques. While lossy data compression does not exactly preserve the original data (as lossless compression does), lossy techniques have an advantage in terms of smaller storage requirements. To preserve the integrity of the scientific simulation data, the effects of lossy data compression on the original data should, at a minimum, not be statistically distinguishable from the natural variability of the climate system, and previous preliminary work with data from CESM has shown this goal to be attainable. However, to ultimately convince climate scientists that it is acceptable to use lossy data compression, we provide climate scientists with access to publicly available climate data that have undergone lossy data compression. In particular, we report on the results of a lossy data compression experiment with output from the CESM Large Ensemble (CESM-LE) Community Project, in which we challenge climate scientists to examine features of the data relevant to their interests, and attempt to identify which of the ensemble members have been compressed and reconstructed. We find that while detecting distinguishing features is certainly possible, the compression effects noticeable in these features are often unimportant or disappear in post-processing analyses. In addition, we perform several analyses that directly compare the original data to the reconstructed data to investigate the preservation, or lack thereof, of specific features critical to climate science. Overall, we conclude that applying lossy data compression to climate simulation data is both advantageous in terms of data reduction and generally acceptable in terms of effects on scientific results.