Collaborative Research: EarthCube Data Capabilities--Jupyter Meets the Earth: Enabling Discovery in Geoscience through Interactive Computing at Scale

协作研究：EarthCube 数据能力——Jupyter 遇见地球：通过大规模交互式计算实现地球科学发现

• 批准号: 1928406
• 负责人: Fernando Perez
• 金额: $ 171.26万
• 依托单位: University of California-Berkeley
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-09-01 至 2023-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1928406&HistoricalAwards=false
• 关键词: Collaborative; Research; EarthCube; Data; Capabilities

Earth science research is being reshaped by the availability of increasing amounts and variety of data, combined with ever more refined and computationally demanding models. This transition to a data-rich world offers immense opportunities for transformative scientific discoveries, but also presents new challenges to researchers: exploring these vast stores of data and combining them with complex models to make discoveries and novel predictions is technically challenging, requiring data management and computational expertise distinct from that of many Earth scientists. This project will develop novel tools to help Earth scientists seamlessly access and interact with extremely large data sets and powerful computational resources, in an environment that supports the lifecycle of research ideas from the scientist to the public. Specifically, this new effort builds upon the foundations of Project Jupyter, which provides tools for interactive computing, and partners with the Pangeo project that develops open tools and fosters a community of Big Data geoscientists. In this project, researchers will build new tools for interactive access to and exploration of data and models, driven by three specific problems in geoscience: the analysis of global climate models, the hydrology of watersheds, and the modeling of the subsurface of the Earth based on measurements of electric and magnetic fields. The project will advance technologies that empower multiple communities of researchers, both in Earth science and beyond. Tools from Project Jupyter are being used worldwide in research, education, industry, government, and media, including in the groundbreaking observation of gravitational waves by the Laser Interferometer Gravitational-Wave Observatory collaboration and the first direct image of a black hole made by the Event Horizon Telescope. The outcomes of the project will be freely available to the public as Open Source software. The project will use geoscience use-cases in hydrology, climate science, and geophysics to drive the advancement of computational technologies for interactive geoscience research involving very large datasets and computationally complex models. These use-cases require High Performance Computing facilities or distributed computing in the cloud, and highlight the need for capabilities to: (1) handle big data such as the World Climate Research Program's Coupled Model Intercomparison Project's 6th release, expected to exceed 18 petabytes in size, (2) integrate data over variable spatial and temporal scales, including streamflow forecasts with sensor-based observations of discharge and hydrometeorological forcing factors, such as precipitation, temperature, relative humidity, and snow-water equivalent, (3) perform large-scale, parallelized computations that combine the solution of partial differential equations with numerical optimization to construct 3D models of the subsurface in a geophysical inversion of electromagnetic data. The project team is an interdisciplinary collaboration that brings together software developers, geoscientists, and statisticians to advance the state of data science in the geosciences. The researchers will follow a user-centered design approach that Project Jupyter has successfully applied for over 15 years, using concrete use-cases to constrain and prioritize software development and ensure that all resulting features have direct scientific relevance. The key software goals of the project are to: (a) improve access to data sources and data catalogs by exposing them to users in the same Jupyter interface where they conduct their computational work, (b) empower researchers to seamlessly utilize and combine cloud and high performance computing resources, (c) accelerate research by simplifying the process for scientists to create and deploy custom, interactive applications for their research questions, and (d) facilitate dissemination of research findings to decision-makers, stakeholders, and the general public. To achieve these, the project will advance three key Jupyter technologies: JupyterLab, Jupyter Widgets and JupyterHub. JupyterLab is an extensible interface that provides access to data, computation, and visualization. Jupyter Widgets provide easy-to-use tools for researchers to create rich graphical user interfaces for data analysis. JupyterHub is a tool for deploying computational web-based interfaces on shared infrastructure, such as the cloud or High Performance Computing centers. By working on three concrete geoscience problems the researchers will advance the state of the art in their respective fields, yet in their implementation within the open Jupyter ecosystem they will ensure that their solutions are generalizable to other scientific domains.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

地球科学研究正在被不断增加的数据量和种类的可用性所重塑，再加上更加精细和计算要求更高的模型。这种向数据丰富世界的转变为变革性的科学发现提供了巨大的机会，但也给研究人员带来了新的挑战：探索这些巨大的数据存储，并将它们与复杂的模型相结合，以做出发现和新的预测，这在技术上是具有挑战性的，需要与许多地球科学家不同的数据管理和计算专业知识。该项目将开发新颖的工具，以帮助地球科学家在一个支持从科学家到公众的研究思想生命周期的环境中无缝访问和交互超大数据集和强大的计算资源。具体来说，这项新工作建立在Jupyter项目的基础上，该项目为交互式计算提供工具，并与Pangeo项目合作开发开放工具，培养大数据地球科学家社区。在这个项目中，研究人员将建立新的工具，用于交互式访问和探索数据和模型，这是由地球科学中的三个具体问题驱动的：全球气候模型分析、流域水文和基于电场和磁场测量的地球地下建模。该项目将推动技术的发展，使地球科学和其他领域的多个研究人员群体获得能力。“木星计划”的工具正在世界范围内的研究、教育、工业、政府和媒体中使用，包括激光干涉仪引力波天文台合作项目对引力波的开创性观测，以及事件视界望远镜拍摄的第一张黑洞直接图像。该项目的成果将作为开源软件免费提供给公众。该项目将利用水文学、气候科学和地球物理学中的地球科学用例，推动计算技术的进步，用于涉及非常大的数据集和计算复杂模型的交互式地球科学研究。这些用例需要高性能计算设施或云中的分布式计算，并强调对以下功能的需求：(1)处理大数据，如世界气候研究计划耦合模式比对项目第6次发布，预计规模将超过18 pb；(2)整合不同时空尺度的数据，包括流量预报与基于传感器的流量和水文气象强迫因子观测，如降水、温度、相对湿度和雪水当量；将偏微分方程的解与数值优化相结合的并行计算，在电磁数据的地球物理反演中构建地下三维模型。项目团队是一个跨学科的协作，将软件开发人员、地球科学家和统计学家聚集在一起，以推进地球科学中的数据科学状态。研究人员将遵循以用户为中心的设计方法，该方法在Jupyter项目中已经成功应用了15年，使用具体的用例来约束和优先考虑软件开发，并确保所有结果特征都具有直接的科学相关性。项目的主要软件目标是：(a)通过在用户进行计算工作的同一个Jupyter界面中向用户公开数据源和数据目录，从而改善对数据源和数据目录的访问；(b)使研究人员能够无缝地利用和结合云和高性能计算资源；(c)通过简化科学家为其研究问题创建和部署自定义交互式应用程序的过程来加速研究；和普通大众。为了实现这些目标，该项目将推进Jupyter的三项关键技术：JupyterLab、Jupyter Widgets和JupyterHub。JupyterLab是一个可扩展的接口，提供对数据、计算和可视化的访问。Jupyter Widgets为研究人员提供了易于使用的工具，用于创建丰富的图形用户界面进行数据分析。JupyterHub是一个用于在共享基础设施（如云或高性能计算中心）上部署基于web的计算接口的工具。通过研究三个具体的地球科学问题，研究人员将在各自的领域推进最先进的技术，但在开放的木星生态系统中实施，他们将确保他们的解决方案可推广到其他科学领域。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

GLAcier Feature Tracking testkit (GLAFT): a statistically and physically based framework for evaluating glacier velocity products derived from optical satellite image feature tracking

GLAcier 特征跟踪测试套件 (GLAFT)：一个基于统计和物理的框架，用于评估源自光学卫星图像特征跟踪的冰川速度产品

• DOI: 10.5194/tc-17-4063-2023
• 发表时间: 2023
• 期刊: The Cryosphere
• 作者: Zheng, Whyjay;Bhushan, Shashank;Van Wyk De Vries, Maximillian;Kochtitzky, William;Shean, David;Copland, Luke;Dow, Christine;Jones-Ivey, Renette;Pérez, Fernando
• 通讯作者: Pérez, Fernando

Towards Interactive, Reproducible Analytics at Scale on HPC Systems

在 HPC 系统上实现大规模交互式、可重复分析

• DOI: 10.1109/urgenthpc51945.2020.00011
• 发表时间: 2020
• 期刊: 2020 IEEE/ACM HPC for Urgent Decision Making (UrgentHPC
• 作者: Cholia, Shreyas;Heagy, Lindsey;Henderson, Matthew;Paine, Drew;Hays, Jon;Bianchi, Ludovico;Ghoshal, Devarshi;Perez, Fernando;Ramakrishnan, Lavanya
• 通讯作者: Ramakrishnan, Lavanya

Jupyter: Thinking and Storytelling With Code and Data

• DOI: 10.1109/mcse.2021.3059263
• 发表时间: 2021-03-01
• 期刊: COMPUTING IN SCIENCE & ENGINEERING
• 影响因子: 2.1
• 作者: Granger, Brian E.;Perez, Fernando
• 通讯作者: Perez, Fernando

Glacier geometry and flow speed determine how Arctic marine-terminating glaciers respond to lubricated beds

冰川几何形状和流速决定北极海洋终止冰川对润滑床的反应

• DOI: 10.5194/tc-16-1431-2022
• 发表时间: 2022
• 期刊: The Cryosphere
• 作者: Zheng, Whyjay

HydroBench: Jupyter supported reproducible hydrological model benchmarking and diagnostic tool

• DOI: 10.3389/feart.2022.884766
• 发表时间: 2022-09
• 作者: E. Moges;B. Ruddell;Liang Zhang;J. Driscoll;P. Norton;Fernando Pérez;L. Larsen