Virtual Data Set Services Enabling New Science at NSF Facilities

美国国家科学基金会支持的Daniel K. Inouye太阳望远镜（DKIST）、国家大气研究中心（NCAR）和国家生态观测站网络（NEON）等科学设施收集了大量有关我们生活的世界的宝贵数据。这些数据可以用于科学和社会效益：在太阳的行为和磁场以及我们不断变化的环境方面取得突破性发现；提高强风暴和破坏性野火预报的速度和准确性；预测太阳耀斑对电力系统的破坏；还有很多其他的目的。在这些重要的科学数据得到有效利用之前，它们必须迅速、有效和可靠地交付给需要它们的人。研究人员需要交互式社区访问位于大数据档案中的难以获取的数据。由于NCAR、DKIST和NEON数据档案无法切实提供所有分析所需的计算资源，终端用户科学家需要能够定义、导航、下载和分析数据子集。目前基于网络的工具还不能胜任这些任务。NSF设施项目的虚拟数据集服务将通过开发组织、包装和快速传输数据的新方法来解决这一挑战。一个关键的创新将是开发用于定义、共享和操作“虚拟数据集”的方法，这些数据集是为特定目的从大量科学设施中“动态”提取的数据集合。研究人员可以定义虚拟数据集，就像购物者在网上“购物车”中组装产品一样。一旦定义，虚拟数据集就可以转移到远程计算机上进行分析，与同事共享，或为未来的项目扩展。我们将开发新的服务来(a)实现对科学设施的千万亿级数据档案中的虚拟数据集的定义、导航和选择性访问，以及(b)确保千万亿级数据档案和其他位置之间的整个数据集或数据子集的可靠、自动化、高效和安全复制和访问，包括最终用户计算机和旨在加速社区成员数据访问的远程镜像。这些新服务将建立在Globus平台之上，该平台已经在NCAR的研究数据档案（RDA）和许多其他研究数据中心大量使用。最终目标是与DKIST、NEON和NCAR/RDA合作，将新服务集成到操作系统中。结果将与太阳物理学、大气科学和生态学研究人员合作，在要求苛刻的科学应用背景下进行评估。该项目由计算机和信息科学与工程理事会的高级网络基础设施办公室支持。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Scientific facilities supported by the National Science Foundation such as the Daniel K. Inouye Solar Telescope (DKIST), National Center for Atmospheric Research (NCAR), and National Ecological Observatory Network (NEON) collect enormous quantities of valuable data about the world in which we live. These data can be used for scientific and societal benefit: to make breakthrough discoveries about sun's behavior and magnetic field, and our changing environment; to improve the speed and accuracy of forecasting of severe storms and destructive wildfires; to predict disruptions to electrical systems from solar flares; and many other purposes. Before such vital scientific data can be used effectively, they must be delivered rapidly, efficiently, and reliably to the people who need them. Researchers need interactive community access to hard-to-obtain data located in large data archives. Because the NCAR, DKIST, and NEON data archives cannot feasibly provide the computing resources needed for all analyses, end-user scientists need to be able to define, navigate, download, and analyze data subsets. Current web-based tools are not up to these tasks. The Virtual Data Set Services Enabling New Science at NSF Facilities project will tackle this challenge by developing new methods for organizing, packaging, and rapidly transporting data. A key innovation will be the development of methods for defining, sharing, and manipulating "virtual data sets," data collections extracted "on the fly" from the vast holdings of scientific facilities for a specific purpose. A researcher may define a virtual data set much as a shopper assembles products in an online "shopping cart." Once defined, a virtual data set can then be transferred to a remote computer for analysis, shared with colleagues, or extended for future projects. We will develop new services to (a) enable definition of, navigation over, and selective access to virtual data sets from petascale data archives of the scientific facilities, and (b) ensure reliable, automated, efficient, and secure replication and access of entire data sets or data subsets between a petascale data archive and other locations, to include both end user computers and remote mirrors intended to accelerate data access by community members. These new services will be constructed on top of the Globus platform, already heavily used within NCAR's Research Data Archive (RDA) and many other research data centers. The ultimate aim is to integrate the new services into operational systems in collaboration with DKIST, NEON, and NCAR/RDA. The results will be evaluated in the context of demanding science applications in partnership with solar physics, atmospheric science, and ecology researchers.This project is supported by the Office of Advanced Cyberinfrastructure in the Directorate for Computer and Information Science and Engineering.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.