Collaborative Research: Elements: Advancing Data Science and Analytics for Water (DSAW)

水文和水资源领域的科学挑战，如理解气候变化的影响、人口增长和土地利用变化对供水的可持续性、水文变化对生态系统和人类的影响等，都是越来越多的数据密集型问题。环境科学家为研究水文系统而产生的大量数据需要先进的软件工具来进行有效的数据可视化、分析和建模。科学家花费大量时间访问、组织和准备用于分析的数据集，这可能成为有效分析的障碍，并阻碍科学探究和进步。该项目将开发新的软件，以提高科学家在水文和水资源领域应用先进数据可视化和分析方法（统称为“数据科学”方法）的能力。该项目将促进标准化的软件工具和数据格式，以帮助科学家提高他们进行的分析的一致性、可共享性和可重复性——所有这些对于建立对科学结果的信任都很重要。该项目开发的软件将使数据加载和分析组织更容易，减少科学家选择合适的数据结构和编写计算机代码来读取和解析数据所花费的时间。它将使用户能够自动从HydroShare系统中检索数据，该系统是一个水文领域的数据存储库，也可以从重要的国家水数据源中检索数据，如美国地质调查局的国家水信息系统。该软件将自动将这些来源的数据加载到针对特定科学数据类型的标准化和高性能数据结构中，并集成了可视化、分析和水文和水资源领域科学家常用的其他数据科学功能。该项目还将减少水科学家创建计算环境的技术负担，在计算环境中，通过安装和维护由大学联盟开发的Python包来执行他们的分析，促进水文科学公司（CUAHSI） hydroshare链接的JupyterHub环境。最后，该项目将通过制作一套基于真实水数据科学应用的教育模块来演示软件的功能和使用，这些模块为向社区提供软件并促进其在课堂和研究环境中的使用提供了具体机制。水领域的科学和相关管理挑战本质上是多学科的，需要综合来自多个领域的多种类型的数据。水科学家执行的许多数据操作、可视化和分析任务都很困难，因为：(1)数据集变得越来越大、越来越复杂；(2)通用数据类型的标准数据格式并不总是达成一致，即使达成一致，也不总是映射到有效的结构，以便在分析环境中进行可视化和/或分析；(3)水科学家通常缺乏数据密集型科学方法方面的培训，这将使他们能够使用新的和现有的工具来有效地处理大型和复杂的数据集。该项目将通过开发：(1)在面向对象的Python语言和基于标准文件、数据和内容类型的分析环境中，将常见的与水相关的数据类型映射到高性能数据结构的高级对象数据模型，该数据类型是由促进水文科学大学联盟（CUAHSI） HydroShare系统建立的；(2)两个新的Python包，使用户能够编写Python代码来自动检索所需的水数据，将其加载到项目中设计的对象数据模型指定的高性能内存对象中，并以可重复的方式执行分析，可以共享，协作，并正式发布以供重用。该项目将使用特定领域的数据科学应用程序来演示如何将新的Python包与现有Python包（如Pandas、numpy和scikit-learn）强大的数据科学功能配对，以在云和桌面环境中开发高级分析工作流。该项目旨在扩展HydroShare数据和模型存储库的数据访问、协作和存档能力，并促进其作为可复制水数据科学平台的使用。该项目还旨在克服与访问、组织和准备数据集相关的障碍，以进行数据科学密集型分析。克服这些障碍将有助于在水文和水资源领域转变科学调查和推进数据科学方法的应用。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Scientific challenges in hydrology and water resources such as understanding impacts of variable climate, sustainability of water supply with population growth and land use change, and impacts of hydrologic change on ecosystems and humans are increasingly data intensive. The volume of data produced by environmental scientists to study hydrologic systems requires advanced software tools for effective data visualization, analysis, and modeling. Scientists spend much of their time accessing, organizing, and preparing datasets for analyses, which can be a barrier to efficient analyses and hinders scientific inquiries and advances. This project will develop new software that will enhance scientists' ability to apply advanced data visualization and analysis methods (collectively referred to as "data science" methods) in the hydrology and water resources domain. The project will promote standardized software tools and data formats to help scientists enhance the consistency, share-ability, and reproducibility of the analyses they perform - all of which are important in building trust in scientific results. The software developed in the project will make data loading and organization for analysis easier, reducing the time spent by scientists in choosing appropriate data structures and writing computer code to read and parse data. It will enable users to automatically retrieve data from the HydroShare system, which is a hydrology domain data repository, as well as from important national water data sources like the United States Geological Survey's National Water Information System. The software will automatically load data from these sources into standardized and high performance data structures targeted to specific scientific data types and that integrate with visualization, analysis, and other data science capabilities commonly used by scientists in the hydrology and water resources domains. The project will also reduce the technical burden for water scientists associated with creating a computational environment within which to execute their analyses by installing and maintaining the Python packages developed within the Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI) HydroShare-linked JupyterHub environment. Finally, the project will demonstrate the functionality and use of the software by producing a set of educational modules based on real water-data science applications that provide a specific mechanism for delivering the software to the community and promoting its use in classroom and research environments.Scientific and related management challenges in the water domain are inherently multi-disciplinary, requiring synthesis of data of multiple types from multiple domains. Many data manipulation, visualization, and analysis tasks performed by water scientists are difficult because (1) datasets are becoming larger and more complex; (2) standard data formats for common data types are not always agreed upon, and, when they are, they are not always mapped to an efficient structure for visualization and/or analysis within an analytical environment; and (3) water scientists generally lack training in data intensive scientific methods that would enable them to use new and existing tools to efficiently tackle large and complex datasets. This project will advance Data Science and Analytics for Water (DSAW) by developing: (1) an advanced object data model that maps common water-related data types to high performance data structures within the object-oriented Python language and analytical environment based upon standard file, data, and content types established by the Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI) HydroShare system; (2) two new Python packages that enable users to write Python code for automating retrieval of desired water data, loading it into high performance memory objects specified by the object data model designed in the project, and performing analysis in a reproducible way that can be shared, collaborated around, and formally published for reuse. The project will use domain-specific data science applications to demonstrate how the new Python packages can be paired with the powerful data science capabilities of existing Python packages like Pandas, numpy, and scikit-learn to develop advanced analytical workflows within cloud and desktop environments. The project aims to extend the data access, collaboration, and archival capabilities of the HydroShare data and model repository and promote its use as a platform for reproducible water-data science. The project also aims to overcome barriers associated with accessing, organizing, and preparing datasets for data science intensive analyses. Overcoming these barriers will be an enabler for transforming scientific inquiries and advancing application of data science methods in the hydrology and water resources domains.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.