Collaborative Research: Elements: Advancing Data Science and Analytics for Water (DSAW)

水文学和水资源方面的科学挑战，如了解气候变化的影响、供水随人口增长和土地利用变化的可持续性、以及水文变化对生态系统和人类的影响等方面的数据越来越密集。环境科学家为研究水文系统而产生的大量数据需要先进的软件工具，以实现有效的数据可视化、分析和建模。科学家花费大量时间访问、组织和准备用于分析的数据集，这可能是有效分析的障碍，并阻碍科学查询和进展。该项目将开发新的软件，提高科学家在水文和水资源领域应用先进的数据可视化和分析方法(统称为“数据科学”方法)的能力。该项目将促进标准化的软件工具和数据格式，以帮助科学家提高他们执行的分析的一致性、共享性和重复性--所有这些都对建立对科学结果的信任非常重要。该项目开发的软件将使数据加载和组织分析变得更容易，减少科学家选择适当的数据结构和编写计算机代码读取和解析数据所花费的时间。它将使用户能够自动从水文领域数据储存库该软件将自动将这些来源的数据加载到针对特定科学数据类型的标准化和高性能数据结构中，并与水文和水资源领域的科学家通常使用的可视化、分析和其他数据科学功能相结合。该项目还将通过安装和维护在水文科学促进大学联盟(CUAHSI)中开发的Python包来减轻水科学家在创建执行其分析的计算环境方面的技术负担。最后，该项目将展示该软件的功能和用途，根据实际的水数据科学应用程序制作一套教育模块，为向社区提供软件并促进其在课堂和研究环境中的使用提供具体机制。水领域的科学和相关管理挑战本质上是多学科的，需要综合多个领域的多种类型的数据。水科学家执行的许多数据处理、可视化和分析任务很困难，原因是：(1)数据集变得越来越大和复杂；(2)常见数据类型的标准数据格式并不总是得到一致同意，而且当它们达成一致时，它们并不总是被映射到分析环境中用于可视化和/或分析的有效结构；以及(3)水科学家通常缺乏数据密集型科学方法方面的培训，这将使他们能够使用新的和现有的工具来有效地处理大型和复杂的数据集。该项目将通过以下方式推进水数据科学和分析(DSAW)：(1)高级对象数据模型，它在面向对象的Python语言和分析环境中将常见的与水相关的数据类型映射到高性能数据结构，该语言和分析环境基于由大学水文科学促进会(CUAHSI)建立的标准文件、数据和内容类型(CUAHSI)水文共享系统；(2)两个新的Python包，使用户能够编写用于自动检索所需水数据的Python代码，将其加载到项目中设计的对象数据模型指定的高性能内存对象中，并以可重复的方式执行分析，这些分析可以共享、协作并正式发布以供重用。该项目将使用特定于领域的数据科学应用程序来演示如何将新的Python包与现有的Python包(如Pandas、NumPy和SCRICKIT)的强大数据科学功能结合在一起-学习在云和桌面环境中开发高级分析工作流。该项目旨在扩大水共享数据和模型库的数据访问、协作和存档能力，并促进将其用作可再生水数据科学的平台。该项目还旨在克服与访问、组织和准备用于数据科学密集型分析的数据集相关的障碍。克服这些障碍将促进水文和水资源领域的科学研究和推动数据科学方法的应用。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Scientific challenges in hydrology and water resources such as understanding impacts of variable climate, sustainability of water supply with population growth and land use change, and impacts of hydrologic change on ecosystems and humans are increasingly data intensive. The volume of data produced by environmental scientists to study hydrologic systems requires advanced software tools for effective data visualization, analysis, and modeling. Scientists spend much of their time accessing, organizing, and preparing datasets for analyses, which can be a barrier to efficient analyses and hinders scientific inquiries and advances. This project will develop new software that will enhance scientists' ability to apply advanced data visualization and analysis methods (collectively referred to as "data science" methods) in the hydrology and water resources domain. The project will promote standardized software tools and data formats to help scientists enhance the consistency, share-ability, and reproducibility of the analyses they perform - all of which are important in building trust in scientific results. The software developed in the project will make data loading and organization for analysis easier, reducing the time spent by scientists in choosing appropriate data structures and writing computer code to read and parse data. It will enable users to automatically retrieve data from the HydroShare system, which is a hydrology domain data repository, as well as from important national water data sources like the United States Geological Survey's National Water Information System. The software will automatically load data from these sources into standardized and high performance data structures targeted to specific scientific data types and that integrate with visualization, analysis, and other data science capabilities commonly used by scientists in the hydrology and water resources domains. The project will also reduce the technical burden for water scientists associated with creating a computational environment within which to execute their analyses by installing and maintaining the Python packages developed within the Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI) HydroShare-linked JupyterHub environment. Finally, the project will demonstrate the functionality and use of the software by producing a set of educational modules based on real water-data science applications that provide a specific mechanism for delivering the software to the community and promoting its use in classroom and research environments.Scientific and related management challenges in the water domain are inherently multi-disciplinary, requiring synthesis of data of multiple types from multiple domains. Many data manipulation, visualization, and analysis tasks performed by water scientists are difficult because (1) datasets are becoming larger and more complex; (2) standard data formats for common data types are not always agreed upon, and, when they are, they are not always mapped to an efficient structure for visualization and/or analysis within an analytical environment; and (3) water scientists generally lack training in data intensive scientific methods that would enable them to use new and existing tools to efficiently tackle large and complex datasets. This project will advance Data Science and Analytics for Water (DSAW) by developing: (1) an advanced object data model that maps common water-related data types to high performance data structures within the object-oriented Python language and analytical environment based upon standard file, data, and content types established by the Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI) HydroShare system; (2) two new Python packages that enable users to write Python code for automating retrieval of desired water data, loading it into high performance memory objects specified by the object data model designed in the project, and performing analysis in a reproducible way that can be shared, collaborated around, and formally published for reuse. The project will use domain-specific data science applications to demonstrate how the new Python packages can be paired with the powerful data science capabilities of existing Python packages like Pandas, numpy, and scikit-learn to develop advanced analytical workflows within cloud and desktop environments. The project aims to extend the data access, collaboration, and archival capabilities of the HydroShare data and model repository and promote its use as a platform for reproducible water-data science. The project also aims to overcome barriers associated with accessing, organizing, and preparing datasets for data science intensive analyses. Overcoming these barriers will be an enabler for transforming scientific inquiries and advancing application of data science methods in the hydrology and water resources domains.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.