Collaborative Research: EarthCube Data Capabilities--Jupyter Meets the Earth: Enabling Discovery in Geoscience through Interactive Computing at Scale

地球科学研究正在重塑，因为越来越多的数据和种类越来越多的数据，以及越来越精细和对计算要求更高的模型。这种向数据丰富的世界的转变为变革性的科学发现提供了巨大的机会，但也给研究人员带来了新的挑战：探索这些海量存储的数据，并将它们与复杂的模型结合起来，以做出发现和新颖的预测，这在技术上具有挑战性，需要有别于许多地球科学家的数据管理和计算专业知识。该项目将开发新的工具，帮助地球科学家在一个支持从科学家到公众的研究想法的生命周期的环境中，无缝地访问极大的数据集和强大的计算资源，并与之互动。具体地说，这一新的努力建立在为交互计算提供工具的Jupyter项目的基础上，并与开发开放工具并培育大数据地球科学家社区的Pangeo项目合作。在这个项目中，研究人员将在地球科学中的三个具体问题的推动下，建立新的工具，以交互方式获取和探索数据和模型：全球气候模型的分析、流域的水文学以及基于电场和磁场测量的地球次表层建模。该项目将推进技术进步，为地球科学和其他领域的多个研究人员社区提供能力。Jupyter项目的工具正在世界各地的研究、教育、工业、政府和媒体中使用，包括由激光干涉仪引力波天文台合作进行的开创性引力波观测，以及由Event Horizon望远镜拍摄的第一张黑洞直接图像。该项目的成果将以开放源码软件的形式向公众免费提供。该项目将使用水文学、气候科学和地球物理学中的地球科学用例来推动互动地球科学研究的计算技术的进步，涉及非常大的数据集和计算复杂的模型。这些用例需要高性能计算设施或云中的分布式计算，并突出了对以下能力的需求：(1)处理大数据，例如世界气候研究计划的耦合模式比较项目的第6版，预计大小将超过18 PB；(2)在不同的空间和时间尺度上集成数据，包括流量预报和基于传感器的流量和水文气象强迫因素的观测，如降水、温度、相对湿度和雪水当量；(3)执行大规模、并行计算，将偏微分方程解与数值优化相结合，以构建电磁数据地球物理反演中的地下3D模型。该项目团队是一个跨学科协作，将软件开发人员、地球科学家和统计学家聚集在一起，以推动地球科学中数据科学的发展。研究人员将遵循Project Jupyter成功应用了15年多的以用户为中心的设计方法，使用具体的用例来约束软件开发并确定其优先顺序，并确保所有产生的功能都具有直接的科学相关性。该项目的主要软件目标是：(A)通过在进行计算工作的同一Jupyter界面中向用户公开数据源和数据目录来改善对它们的访问，(B)使研究人员能够无缝地利用和组合云和高性能计算资源，(C)通过简化科学家为其研究问题创建和部署定制的交互式应用程序的过程来加速研究，以及(D)促进向决策者、利益攸关方和公众传播研究成果。为了实现这些，该项目将推进Jupyter的三项关键技术：JupyterLab、Jupyter Widget和JupyterHub。JupyterLab是一个可扩展的接口，提供对数据、计算和可视化的访问。Jupyter小部件为研究人员提供了易于使用的工具，以创建用于数据分析的丰富图形用户界面。JupyterHub是一个工具，用于在共享基础设施(如云或高性能计算中心)上部署基于Web的计算界面。通过研究三个具体的地球科学问题，研究人员将推进各自领域的最新技术，但在开放的Jupyter生态系统中实施时，他们将确保他们的解决方案可推广到其他科学领域。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Earth science research is being reshaped by the availability of increasing amounts and variety of data, combined with ever more refined and computationally demanding models. This transition to a data-rich world offers immense opportunities for transformative scientific discoveries, but also presents new challenges to researchers: exploring these vast stores of data and combining them with complex models to make discoveries and novel predictions is technically challenging, requiring data management and computational expertise distinct from that of many Earth scientists. This project will develop novel tools to help Earth scientists seamlessly access and interact with extremely large data sets and powerful computational resources, in an environment that supports the lifecycle of research ideas from the scientist to the public. Specifically, this new effort builds upon the foundations of Project Jupyter, which provides tools for interactive computing, and partners with the Pangeo project that develops open tools and fosters a community of Big Data geoscientists. In this project, researchers will build new tools for interactive access to and exploration of data and models, driven by three specific problems in geoscience: the analysis of global climate models, the hydrology of watersheds, and the modeling of the subsurface of the Earth based on measurements of electric and magnetic fields. The project will advance technologies that empower multiple communities of researchers, both in Earth science and beyond. Tools from Project Jupyter are being used worldwide in research, education, industry, government, and media, including in the groundbreaking observation of gravitational waves by the Laser Interferometer Gravitational-Wave Observatory collaboration and the first direct image of a black hole made by the Event Horizon Telescope. The outcomes of the project will be freely available to the public as Open Source software. The project will use geoscience use-cases in hydrology, climate science, and geophysics to drive the advancement of computational technologies for interactive geoscience research involving very large datasets and computationally complex models. These use-cases require High Performance Computing facilities or distributed computing in the cloud, and highlight the need for capabilities to: (1) handle big data such as the World Climate Research Program's Coupled Model Intercomparison Project's 6th release, expected to exceed 18 petabytes in size, (2) integrate data over variable spatial and temporal scales, including streamflow forecasts with sensor-based observations of discharge and hydrometeorological forcing factors, such as precipitation, temperature, relative humidity, and snow-water equivalent, (3) perform large-scale, parallelized computations that combine the solution of partial differential equations with numerical optimization to construct 3D models of the subsurface in a geophysical inversion of electromagnetic data. The project team is an interdisciplinary collaboration that brings together software developers, geoscientists, and statisticians to advance the state of data science in the geosciences. The researchers will follow a user-centered design approach that Project Jupyter has successfully applied for over 15 years, using concrete use-cases to constrain and prioritize software development and ensure that all resulting features have direct scientific relevance. The key software goals of the project are to: (a) improve access to data sources and data catalogs by exposing them to users in the same Jupyter interface where they conduct their computational work, (b) empower researchers to seamlessly utilize and combine cloud and high performance computing resources, (c) accelerate research by simplifying the process for scientists to create and deploy custom, interactive applications for their research questions, and (d) facilitate dissemination of research findings to decision-makers, stakeholders, and the general public. To achieve these, the project will advance three key Jupyter technologies: JupyterLab, Jupyter Widgets and JupyterHub. JupyterLab is an extensible interface that provides access to data, computation, and visualization. Jupyter Widgets provide easy-to-use tools for researchers to create rich graphical user interfaces for data analysis. JupyterHub is a tool for deploying computational web-based interfaces on shared infrastructure, such as the cloud or High Performance Computing centers. By working on three concrete geoscience problems the researchers will advance the state of the art in their respective fields, yet in their implementation within the open Jupyter ecosystem they will ensure that their solutions are generalizable to other scientific domains.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.