Collaborative Research: EarthCube Data Capabilities--Jupyter Meets the Earth: Enabling Discovery in Geoscience through Interactive Computing at Scale

点击翻译按钮获取中文摘要

Earth science research is being reshaped by the availability of increasing amounts and variety of data, combined with ever more refined and computationally demanding models. This transition to a data-rich world offers immense opportunities for transformative scientific discoveries, but also presents new challenges to researchers: exploring these vast stores of data and combining them with complex models to make discoveries and novel predictions is technically challenging, requiring data management and computational expertise distinct from that of many Earth scientists. This project will develop novel tools to help Earth scientists seamlessly access and interact with extremely large data sets and powerful computational resources, in an environment that supports the lifecycle of research ideas from the scientist to the public. Specifically, this new effort builds upon the foundations of Project Jupyter, which provides tools for interactive computing, and partners with the Pangeo project that develops open tools and fosters a community of Big Data geoscientists. In this project, researchers will build new tools for interactive access to and exploration of data and models, driven by three specific problems in geoscience: the analysis of global climate models, the hydrology of watersheds, and the modeling of the subsurface of the Earth based on measurements of electric and magnetic fields. The project will advance technologies that empower multiple communities of researchers, both in Earth science and beyond. Tools from Project Jupyter are being used worldwide in research, education, industry, government, and media, including in the groundbreaking observation of gravitational waves by the Laser Interferometer Gravitational-Wave Observatory collaboration and the first direct image of a black hole made by the Event Horizon Telescope. The outcomes of the project will be freely available to the public as Open Source software. The project will use geoscience use-cases in hydrology, climate science, and geophysics to drive the advancement of computational technologies for interactive geoscience research involving very large datasets and computationally complex models. These use-cases require High Performance Computing facilities or distributed computing in the cloud, and highlight the need for capabilities to: (1) handle big data such as the World Climate Research Program's Coupled Model Intercomparison Project's 6th release, expected to exceed 18 petabytes in size, (2) integrate data over variable spatial and temporal scales, including streamflow forecasts with sensor-based observations of discharge and hydrometeorological forcing factors, such as precipitation, temperature, relative humidity, and snow-water equivalent, (3) perform large-scale, parallelized computations that combine the solution of partial differential equations with numerical optimization to construct 3D models of the subsurface in a geophysical inversion of electromagnetic data. The project team is an interdisciplinary collaboration that brings together software developers, geoscientists, and statisticians to advance the state of data science in the geosciences. The researchers will follow a user-centered design approach that Project Jupyter has successfully applied for over 15 years, using concrete use-cases to constrain and prioritize software development and ensure that all resulting features have direct scientific relevance. The key software goals of the project are to: (a) improve access to data sources and data catalogs by exposing them to users in the same Jupyter interface where they conduct their computational work, (b) empower researchers to seamlessly utilize and combine cloud and high performance computing resources, (c) accelerate research by simplifying the process for scientists to create and deploy custom, interactive applications for their research questions, and (d) facilitate dissemination of research findings to decision-makers, stakeholders, and the general public. To achieve these, the project will advance three key Jupyter technologies: JupyterLab, Jupyter Widgets and JupyterHub. JupyterLab is an extensible interface that provides access to data, computation, and visualization. Jupyter Widgets provide easy-to-use tools for researchers to create rich graphical user interfaces for data analysis. JupyterHub is a tool for deploying computational web-based interfaces on shared infrastructure, such as the cloud or High Performance Computing centers. By working on three concrete geoscience problems the researchers will advance the state of the art in their respective fields, yet in their implementation within the open Jupyter ecosystem they will ensure that their solutions are generalizable to other scientific domains.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.