Collaborative Proposal: EarthCube Integration: Pangeo: An Open Source Big Data Climate Science Platform

气候、天气和海洋模拟（地球系统模型; ESM）是研究地球系统的重要工具，既提供了对基本动力学的科学见解，也提供了对地球未来的有价值的实用预测。ESM空间分辨率的不断提高使地球系统过程的物理表示更加真实、更加详细，而模拟的统计集合的激增也大大增强了对不确定性和内部变化的理解。与此同时，模拟数据的生成也达到了PB级，这给地球科学研究人员带来了巨大的下游挑战。挖掘ESM输出以获得科学见解的任务现在本身已经成为一个严重的大数据问题。现有的大数据工具无法轻松应用于ESM数据的分析，导致了广泛的地球科学领域的建设危机。这正是地球立方体被设想来解决的问题。该项目将整合一套开源软件工具（“Pangeo平台”），这些工具可以共同处理PB级ESM数据集。此外，将为这些工具编写培训和教育材料，在网上广泛分发，并将其纳入哥伦比亚现有的教育课程。NCAR最后一年的研讨会将有助于向更广泛的社区介绍Pangeo。美国其他气候模拟中心的合作者将鼓励他们的科学家采用和参与Pangeo项目。除了气候和相关领域，多维数值数组在许多科学领域（例如天文学，材料科学，显微镜）中很常见。然而，占主导地位的大数据软件栈（Hadoop）面向基于表格文本的数据结构，无法轻松摄取PB级多维数值数组。因此，拟议的工作有可能改变数据科学本身，通过一种新颖的，高度可扩展的，高度灵活的工具与学科研究人员熟悉的语法来分析这些数据集。核心技术是python包Dask，一个灵活的并行计算库，提供动态任务调度，和XArray，一个包装层Dask数据结构，提供用户友好的元数据跟踪，索引，和可视化。这些工具与netCDF数据集接口并理解CF约定。他们将在大气科学，地表水文学和物理海洋学的四个高影响地球科学用例中发挥作用。学科科学家将为每个用例定义工作流程，并与计算科学家互动，以演示，基准测试和优化软件。由此产生的软件改进将回馈给上游开源项目，确保平台的长期可持续性。最终的结果将是一个强大的新软件工具包，用于气候科学及其他领域。该工具包将增强EarthCube的数据科学方面。这些工具在云上的实施也将进行测试，利用商业云服务提供商和NSF之间的BIGDATA征集协议。

Climate, weather, and ocean simulations (Earth System Models; ESMs) are crucial tools for the study of the Earth system, providing both scientific insight into fundamental dynamics as well as valuable practical predictions about Earth's future. Continuous increases in ESM spatial resolution have led to more realistic, more detailed physical representations of Earth system processes, while the proliferation of statistical ensembles of simulations has greatly enhanced understanding of uncertainty and internal variability. Hand in hand with this progress has come the generation of Petabytes of simulation data, resulting in huge downstream challenges for geoscience researchers. The task of mining ESM output for scientific insights has now itself become a serious Big Data problem. Existing Big Data tools cannot easily be applied to the analysis of ESM data, leading to a building crisis across a wide range of geoscience fields. This is exactly the sort of problem EarthCube was conceived to address. The project will integrate a suite of open-source software tools (the "Pangeo Platform") which together can tackle petabyte-scale ESM datasets. Additionally, training and educational materials for these tools will be developed, distributed widely online, and integrated into existing educational curricula at Columbia. A workshop at NCAR in the final year will help inform the broader community about Pangeo. Collaborators at other US climate modeling centers will encourage adoption and participation in the Pangeo project by their scientists. Beyond climate and related fields, multidimensional numeric arrays are common in many fields of science (e.g. astronomy, materials science, microscopy). However, the dominant Big Data software stack (Hadoop) is oriented towards tabular text-based data structures and cannot easily ingest petabyte scale multidimensional numeric arrays. The proposed work thus has potential to transform Data Science itself, enabling analysis of such datasets via a novel, highly scalable, highly flexible tool with a syntax familiar to disciplinary researchers.The core technologies are the python packages Dask, a flexible parallel computing library which provides dynamic task scheduling, and XArray, a wrapper layer over Dask data structures which provides user-friendly metadata tracking, indexing, and visualization. These tools interface with netCDF datasets and understand CF conventions. They will be brought to bear on four high impact Geoscience Use Cases in atmospheric science, land-surface hydrology, and physical oceanography. Disciplinary scientists will define workflows for each use case and interact with computational scientists to demonstrate, benchmark, and optimize the software. The resulting software improvements will be contributed back to the upstream open source projects, ensuring long-term sustainability of the platform. The end result will be a robust new software toolkit for climate science and beyond. This toolkit will enhance the Data Science aspect of EarthCube. Implementation of these tools on the cloud will also be tested, taking advantage of agreement between commercial cloud service providers and NSF for the BIGDATA solicitation.