Collaborative Research: EarthCube Data Capabilities--Jupyter Meets the Earth: Enabling Discovery in Geoscience through Interactive Computing at Scale

地球科学研究正在被不断增加的数据量和多样性所重塑，再加上越来越精细和计算要求更高的模型。这种向数据丰富的世界的过渡为变革性的科学发现提供了巨大的机会，但也给研究人员带来了新的挑战：探索这些庞大的数据存储并将其与复杂的模型相结合以进行发现和新颖的预测在技术上具有挑战性，需要不同于许多地球科学家的数据管理和计算专业知识。该项目将开发新的工具，帮助地球科学家在支持从科学家到公众的研究思想生命周期的环境中，无缝地访问超大数据集和强大的计算资源并与之交互。具体来说，这项新的努力建立在Project Phoneyter的基础上，该项目提供交互式计算工具，并与Pangeo项目合作，开发开放工具并培养大数据地球科学家社区。在这个项目中，研究人员将建立新的工具，用于交互式访问和探索数据和模型，由地球科学中的三个具体问题驱动：全球气候模型的分析，流域水文，以及基于电场和磁场测量的地球地下建模。 该项目将推进技术，增强地球科学及其他领域多个研究人员社区的能力。来自Project Phyter的工具正在全球范围内用于研究，教育，工业，政府和媒体，包括激光干涉仪引力波天文台合作对引力波的开创性观测以及事件视界望远镜制作的黑洞的第一张直接图像。该项目的成果将作为开源软件免费提供给公众。该项目将利用水文学、气候科学和地球物理学中的地球科学用例，推动涉及超大型数据集和计算复杂模型的交互式地球科学研究的计算技术进步。这些用例需要高性能计算设施或云中的分布式计算，并强调了对以下功能的需求：（1）处理大数据，如世界气候研究计划的耦合模型相互比较项目的第6次发布，预计大小将超过18 PB，（2）整合不同空间和时间尺度的数据，包括通过基于传感器的流量和水文气象强迫因素（如降水、温度、相对湿度和雪水当量）观测进行径流预报，（3）进行大尺度，并行计算，其将偏微分方程的解与数值优化相结合，以在电磁数据的地球物理反演中构造地下的3D模型。该项目团队是一个跨学科的合作，汇集了软件开发人员，地球科学家和统计学家，以推进地球科学中的数据科学状态。研究人员将遵循以用户为中心的设计方法，该方法已经成功应用了15年以上，使用具体的用例来约束和优先考虑软件开发，并确保所有产生的功能具有直接的科学相关性。该项目的主要软件目标是：（a）通过将数据源和数据目录在用户进行计算工作的同一个MySQL接口中暴露给用户，改善对数据源和数据目录的访问，（B）使研究人员能够无缝利用和联合收割机云和高性能计算资源，（c）通过简化科学家创建和部署定制的过程来加速研究，（d）促进向决策者、利益攸关方和公众传播研究成果。为了实现这些目标，该项目将推进三项关键的Cyberter技术：CyberterLab、CyberterWidgets和CyberterHub。QuixyterLab是一个可扩展的接口，提供对数据、计算和可视化的访问。Quixyter Widgets为研究人员提供了易于使用的工具，用于创建用于数据分析的丰富图形用户界面。CyberterHub是一种用于在共享基础设施（如云或高性能计算中心）上部署基于Web的计算接口的工具。通过研究三个具体的地球科学问题，研究人员将推进各自领域的最新技术水平，但在开放式生态系统中的实施，他们将确保他们的解决方案可推广到其他科学领域。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Earth science research is being reshaped by the availability of increasing amounts and variety of data, combined with ever more refined and computationally demanding models. This transition to a data-rich world offers immense opportunities for transformative scientific discoveries, but also presents new challenges to researchers: exploring these vast stores of data and combining them with complex models to make discoveries and novel predictions is technically challenging, requiring data management and computational expertise distinct from that of many Earth scientists. This project will develop novel tools to help Earth scientists seamlessly access and interact with extremely large data sets and powerful computational resources, in an environment that supports the lifecycle of research ideas from the scientist to the public. Specifically, this new effort builds upon the foundations of Project Jupyter, which provides tools for interactive computing, and partners with the Pangeo project that develops open tools and fosters a community of Big Data geoscientists. In this project, researchers will build new tools for interactive access to and exploration of data and models, driven by three specific problems in geoscience: the analysis of global climate models, the hydrology of watersheds, and the modeling of the subsurface of the Earth based on measurements of electric and magnetic fields. The project will advance technologies that empower multiple communities of researchers, both in Earth science and beyond. Tools from Project Jupyter are being used worldwide in research, education, industry, government, and media, including in the groundbreaking observation of gravitational waves by the Laser Interferometer Gravitational-Wave Observatory collaboration and the first direct image of a black hole made by the Event Horizon Telescope. The outcomes of the project will be freely available to the public as Open Source software. The project will use geoscience use-cases in hydrology, climate science, and geophysics to drive the advancement of computational technologies for interactive geoscience research involving very large datasets and computationally complex models. These use-cases require High Performance Computing facilities or distributed computing in the cloud, and highlight the need for capabilities to: (1) handle big data such as the World Climate Research Program's Coupled Model Intercomparison Project's 6th release, expected to exceed 18 petabytes in size, (2) integrate data over variable spatial and temporal scales, including streamflow forecasts with sensor-based observations of discharge and hydrometeorological forcing factors, such as precipitation, temperature, relative humidity, and snow-water equivalent, (3) perform large-scale, parallelized computations that combine the solution of partial differential equations with numerical optimization to construct 3D models of the subsurface in a geophysical inversion of electromagnetic data. The project team is an interdisciplinary collaboration that brings together software developers, geoscientists, and statisticians to advance the state of data science in the geosciences. The researchers will follow a user-centered design approach that Project Jupyter has successfully applied for over 15 years, using concrete use-cases to constrain and prioritize software development and ensure that all resulting features have direct scientific relevance. The key software goals of the project are to: (a) improve access to data sources and data catalogs by exposing them to users in the same Jupyter interface where they conduct their computational work, (b) empower researchers to seamlessly utilize and combine cloud and high performance computing resources, (c) accelerate research by simplifying the process for scientists to create and deploy custom, interactive applications for their research questions, and (d) facilitate dissemination of research findings to decision-makers, stakeholders, and the general public. To achieve these, the project will advance three key Jupyter technologies: JupyterLab, Jupyter Widgets and JupyterHub. JupyterLab is an extensible interface that provides access to data, computation, and visualization. Jupyter Widgets provide easy-to-use tools for researchers to create rich graphical user interfaces for data analysis. JupyterHub is a tool for deploying computational web-based interfaces on shared infrastructure, such as the cloud or High Performance Computing centers. By working on three concrete geoscience problems the researchers will advance the state of the art in their respective fields, yet in their implementation within the open Jupyter ecosystem they will ensure that their solutions are generalizable to other scientific domains.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.