Collaborative Research: Elements: ProDM: Developing A Unified Progressive Data Management Library for Exascale Computational Science

对极端规模模拟和仪器产生的科学数据进行有效管理对于推动科学发现至关重要。由于数据的规模和科学分析的多样化要求，人们越来越需要以渐进的方式管理数据，以便用户可以在减少数据移动和计算的情况下，根据需要传输尽可能多的数据来进行数据分析。然而，在创建强大且可扩展的网络基础设施服务方面投入的努力很少，这些服务将最近以渐进方法进行的算法创新与科学数据分析联系起来，使得科学家无法获得这些能力。该项目旨在制定一个可持续的PRDM框架，支持逐步管理科学数据，以促进其在科学应用中的使用。该项目的成功将提供一种管理和分析数据的新方法，从而实现新的科学研究和新的发现。此外，该项目的成果将作为公开可用的软件提供，以加强研究网络基础设施，促进教育和教学，并扩大对计算机的参与。ProDM的核心是统一可行的渐进表示法和针对现场和事后分析例程的量身定制的开发。特别是，它涉及三项关键活动。首先，将建立一个数据引擎，以统一最先进的渐进式表示法，并为加速器提供便携硬件支持，以及与其他数据管理和分析库的交互软件接口。第二，将开发一个现场引擎，以促进使用累进表示法进行现场数据分析，其中包括重新设计现场语义和调整运行时动态。第三，将开发一个后自组织引擎，以高效地访问累进数据，并提高后自组织数据分析的数据检索性能。ProDM将被部署在校园范围的计算基础设施和领导系统上，用于与来自气候、聚变、分子动力学和其他领域的真实世界科学应用程序进行集成和评估。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Effective management of scientific data produced by extreme-scale simulations and instruments is crucial for advancing scientific discoveries. Due to the scale of data and the diverse requirements of scientific analytics, there is a growing need to manage data in a progressive manner, such that users can stream as much data as they need to carry out their data analytics with reduced data movement and computation. However, little effort has been put into creating robust and scalable cyberinfrastructure services that link the recent algorithmic innovations in progressive methods with scientific data analytics, leaving these capabilities inaccessible to scientists. This project aims to develop a sustainable framework ProDM that supports the progressive management of scientific data to facilitate its use in scientific applications. The success of this project will enable new scientific research and novel findings by providing a new way to manage and analyze data. Furthermore, outcomes of this project will be delivered as publicly available software to enhance research cyberinfrastructure, promote education and teaching, and broaden participation in computing. ProDM is centered upon the unification of viable progressive representations and tailored development for in-situ and post-hoc analytic routines. In particular, it involves three key activities. First, a data engine will be built to unify state-of-the-art progressive representations, and provide portable hardware support for accelerators as well as interoperative software interfaces to other data management and analytic libraries. Second, an in-situ engine will be developed to facilitate the use of progressive representations for in-situ data analytics, which include a redesign of in-situ semantics and adjustment of runtime dynamics. Third, a post-hoc engine will be developed to efficiently access progressive data and improve the performance of data retrieval for post-hoc data analytics. ProDM will be deployed on campus-wide computing infrastructures and leadership systems for integration and evaluation with real-world scientific applications from climate, fusion, molecular dynamics, and beyond.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.