Collaborative Proposal: EarthCube Integration: Pangeo: An Open Source Big Data Climate Science Platform

气候、天气和海洋模拟（地球系统模型；ESMs）是研究地球系统的重要工具，既提供了对基本动力学的科学见解，也提供了对地球未来有价值的实际预测。ESM空间分辨率的持续提高导致了对地球系统过程的更真实、更详细的物理表征，而模拟的统计集合的激增极大地增强了对不确定性和内部变率的理解。与此同时，产生了pb级的模拟数据，给地球科学研究人员带来了巨大的下游挑战。挖掘ESM输出以获得科学见解的任务现在本身已经成为一个严重的大数据问题。现有的大数据工具不能很容易地应用于ESM数据的分析，这导致了广泛的地球科学领域的建筑危机。这正是“地球立方体”所要解决的问题。该项目将集成一套开源软件工具（“Pangeo平台”），这些工具可以一起处理pb级的ESM数据集。此外，将开发这些工具的培训和教育材料，在网上广泛分发，并整合到哥伦比亚大学现有的教育课程中。最后一年在NCAR举办的一个研讨会将帮助更广泛的社区了解Pangeo。其他美国气候模拟中心的合作者将鼓励他们的科学家采用和参与Pangeo项目。除了气候和相关领域，多维数值数组在许多科学领域（如天文学、材料科学、显微镜）也很常见。然而，占主导地位的大数据软件栈（Hadoop）是面向基于表格文本的数据结构的，不能轻易地摄取pb规模的多维数字数组。因此，拟议的工作有可能改变数据科学本身，通过一种新颖的、高度可扩展的、高度灵活的工具来分析这些数据集，该工具具有学科研究人员熟悉的语法。核心技术是python包Dask（一个灵活的并行计算库，提供动态任务调度）和XArray （Dask数据结构的包装层，提供用户友好的元数据跟踪、索引和可视化）。这些工具与netCDF数据集接口，并理解CF约定。他们将承担大气科学、陆地表面水文学和物理海洋学中四个高影响力的地球科学用例。学科科学家将为每个用例定义工作流程，并与计算科学家进行交互，以演示、基准测试和优化软件。由此产生的软件改进将回馈给上游开源项目，确保平台的长期可持续性。最终的结果将是为气候科学和其他领域提供一个强大的新软件工具包。这个工具包将增强EarthCube的数据科学方面。利用商业云服务提供商和NSF就BIGDATA招标达成的协议，这些工具在云上的实施也将进行测试。

Climate, weather, and ocean simulations (Earth System Models; ESMs) are crucial tools for the study of the Earth system, providing both scientific insight into fundamental dynamics as well as valuable practical predictions about Earth's future. Continuous increases in ESM spatial resolution have led to more realistic, more detailed physical representations of Earth system processes, while the proliferation of statistical ensembles of simulations has greatly enhanced understanding of uncertainty and internal variability. Hand in hand with this progress has come the generation of Petabytes of simulation data, resulting in huge downstream challenges for geoscience researchers. The task of mining ESM output for scientific insights has now itself become a serious Big Data problem. Existing Big Data tools cannot easily be applied to the analysis of ESM data, leading to a building crisis across a wide range of geoscience fields. This is exactly the sort of problem EarthCube was conceived to address. The project will integrate a suite of open-source software tools (the "Pangeo Platform") which together can tackle petabyte-scale ESM datasets. Additionally, training and educational materials for these tools will be developed, distributed widely online, and integrated into existing educational curricula at Columbia. A workshop at NCAR in the final year will help inform the broader community about Pangeo. Collaborators at other US climate modeling centers will encourage adoption and participation in the Pangeo project by their scientists. Beyond climate and related fields, multidimensional numeric arrays are common in many fields of science (e.g. astronomy, materials science, microscopy). However, the dominant Big Data software stack (Hadoop) is oriented towards tabular text-based data structures and cannot easily ingest petabyte scale multidimensional numeric arrays. The proposed work thus has potential to transform Data Science itself, enabling analysis of such datasets via a novel, highly scalable, highly flexible tool with a syntax familiar to disciplinary researchers.The core technologies are the python packages Dask, a flexible parallel computing library which provides dynamic task scheduling, and XArray, a wrapper layer over Dask data structures which provides user-friendly metadata tracking, indexing, and visualization. These tools interface with netCDF datasets and understand CF conventions. They will be brought to bear on four high impact Geoscience Use Cases in atmospheric science, land-surface hydrology, and physical oceanography. Disciplinary scientists will define workflows for each use case and interact with computational scientists to demonstrate, benchmark, and optimize the software. The resulting software improvements will be contributed back to the upstream open source projects, ensuring long-term sustainability of the platform. The end result will be a robust new software toolkit for climate science and beyond. This toolkit will enhance the Data Science aspect of EarthCube. Implementation of these tools on the cloud will also be tested, taking advantage of agreement between commercial cloud service providers and NSF for the BIGDATA solicitation.