Collaborative Proposal: EarthCube Integration: Pangeo: An Open Source Big Data Climate Science Platform

气候、天气和海洋模拟(地球系统模型；ESM)是研究地球系统的重要工具，既提供了对基本动力学的科学见解，也提供了对地球未来的有价值的实用预测。ESM空间分辨率的不断提高导致了对地球系统过程的更现实、更详细的物理表示，而统计模拟集合的激增极大地加强了对不确定性和内部变异性的理解。伴随着这一进展，产生了数PB的模拟数据，这给地学研究人员带来了巨大的下游挑战。挖掘ESM输出以获得科学见解的任务本身现在已经成为一个严重的大数据问题。现有的大数据工具不能轻松地应用于ESM数据的分析，导致了广泛的地学领域的构建危机。这正是地球立方计划要解决的问题。该项目将集成一套开源软件工具(“Pangeo平台”)，这些工具可以共同处理PB级的ESM数据集。此外，将开发这些工具的培训和教育材料，在网上广泛分发，并将其整合到哥伦比亚大学现有的教育课程中。最后一年在NCAR举办的研讨会将有助于让更广泛的社区了解Pangeo。美国其他气候建模中心的合作者将鼓励他们的科学家采用和参与Pangeo项目。除了气候和相关领域，多维数字阵列在许多科学领域(如天文学、材料科学、显微镜)中很常见。然而，占主导地位的大数据软件堆栈(Hadoop)面向基于表格文本的数据结构，不能轻松地接收PB级多维数字数组。因此，拟议的工作具有改变数据科学本身的潜力，通过一个新颖的、高度可扩展的、高度灵活的工具来分析此类数据集，其语法对学科研究人员来说是熟悉的。核心技术是PYTHON包DASK，它是一个灵活的并行计算库，提供动态任务调度，以及XArray，它是DASK数据结构上的包装层，提供用户友好的元数据跟踪、索引和可视化。这些工具与NetCDF数据集交互，并了解CF约定。它们将被用于大气科学、陆地水文学和物理海洋学中的四个高影响地球科学使用案例。学科科学家将为每个用例定义工作流，并与计算科学家互动，以演示、基准和优化软件。由此产生的软件改进将回馈给上游开源项目，确保平台的长期可持续性。最终结果将是一个强大的新软件工具包，用于气候科学和其他领域。该工具包将增强EarthCube的数据科学方面。这些工具在云上的实施也将进行测试，利用商业云服务提供商和NSF之间的协议进行BigData招标。

Climate, weather, and ocean simulations (Earth System Models; ESMs) are crucial tools for the study of the Earth system, providing both scientific insight into fundamental dynamics as well as valuable practical predictions about Earth's future. Continuous increases in ESM spatial resolution have led to more realistic, more detailed physical representations of Earth system processes, while the proliferation of statistical ensembles of simulations has greatly enhanced understanding of uncertainty and internal variability. Hand in hand with this progress has come the generation of Petabytes of simulation data, resulting in huge downstream challenges for geoscience researchers. The task of mining ESM output for scientific insights has now itself become a serious Big Data problem. Existing Big Data tools cannot easily be applied to the analysis of ESM data, leading to a building crisis across a wide range of geoscience fields. This is exactly the sort of problem EarthCube was conceived to address. The project will integrate a suite of open-source software tools (the "Pangeo Platform") which together can tackle petabyte-scale ESM datasets. Additionally, training and educational materials for these tools will be developed, distributed widely online, and integrated into existing educational curricula at Columbia. A workshop at NCAR in the final year will help inform the broader community about Pangeo. Collaborators at other US climate modeling centers will encourage adoption and participation in the Pangeo project by their scientists. Beyond climate and related fields, multidimensional numeric arrays are common in many fields of science (e.g. astronomy, materials science, microscopy). However, the dominant Big Data software stack (Hadoop) is oriented towards tabular text-based data structures and cannot easily ingest petabyte scale multidimensional numeric arrays. The proposed work thus has potential to transform Data Science itself, enabling analysis of such datasets via a novel, highly scalable, highly flexible tool with a syntax familiar to disciplinary researchers.The core technologies are the python packages Dask, a flexible parallel computing library which provides dynamic task scheduling, and XArray, a wrapper layer over Dask data structures which provides user-friendly metadata tracking, indexing, and visualization. These tools interface with netCDF datasets and understand CF conventions. They will be brought to bear on four high impact Geoscience Use Cases in atmospheric science, land-surface hydrology, and physical oceanography. Disciplinary scientists will define workflows for each use case and interact with computational scientists to demonstrate, benchmark, and optimize the software. The resulting software improvements will be contributed back to the upstream open source projects, ensuring long-term sustainability of the platform. The end result will be a robust new software toolkit for climate science and beyond. This toolkit will enhance the Data Science aspect of EarthCube. Implementation of these tools on the cloud will also be tested, taking advantage of agreement between commercial cloud service providers and NSF for the BIGDATA solicitation.