EAGER: Online Processing of Data in Large Facilities using National Advanced CyberInfrastructure

开放的大型科研设施是科技工程企业的重要组成部分。这些设施提供共享使用的基础设施、仪器和数据产品，对广大的研究人员和/或教育工作者开放。目前的设施提供了越来越多的数据和数据产品，这些数据和产品有可能在广泛的科学和工程领域提供新的见解。然而，虽然这些设施提供了对数据和数据产品的可靠和普遍的访问，但用户通常必须下载感兴趣的数据并使用本地资源处理它们。因此，要将这些数据和数据产品转化为洞察力，就需要本地访问强大的计算、存储和网络资源。另一方面，美国国家科学基金会高级网络基础设施（ACI）作为计算和数据支持科学和工程的开放平台，正发挥着越来越重要的作用，并可以提供必要的功能，允许广泛的用户社区有效地处理大型设施中的数据。然而，尽管有明显的互补，大型科学机构和NSF ACI在很大程度上仍然是脱节的。因此，用户被迫积极参与将数据从大型设施转移到本地计算资源或NSF ACI的过程。因此，这种数据传递模式变得低效，并且限制了以自动方式处理数据所具有的潜在效用。这项研究的结果可以通过改善数据的可访问性以及科学家与数据源和计算基础设施的交互方式，对科学和工程界产生重大影响。将国家ACI和大型科学设施结合在一起，将使科学的获取民主化，并改善nsf资助的基础设施的影响。这对于资源有限且没有高带宽互联网连接到学术/研究网络的小型公共机构尤其重要。人力资源的发展，包括培训学生、研究人员和软件专业人员，以及向少数民族和代表性不足的群体伸出援手，将是这一努力的一个组成部分。该项目使用一个开放的存储库向社区传播研究论文、原型实现和相关的数据产品。该项目的目标是探索nsf资助的ACI，如极限科学与工程发现环境（XSEDE），如何以自动化的方式与大型设施集成，特别是海洋观测倡议（OOI），以支持端到端的用户工作流程。具体来说，我们建议启用工作流，当触发时可以无缝地编排整个数据到发现管道。这包括在OOI网络基础设施上执行查询（可能基于感兴趣的事件的发生），使用高带宽互连（如Internet2）将数据流式传输到适当的ACI设施，以便将这些数据放置在接近计算/分析资源（例如，XSEDE JetStream）的位置，然后启动建模和分析过程，将这些数据转换为见解。通过这种方式，该项目将利用通常连接这些设施的高性能网络来支持数据移动，并使用最先进的高性能系统处理这些数据。

Open, large-scale scientific facilities are an essential part of science and engineering enterprise. These facilities provide shared-use infrastructure, instrumentation, and data products that are openly accessible to a broad community of researchers and/or educators. Current facilities provide increasing volumes of data and data products that have the potential to deliver new insights in a wide range of science and engineering domains. However, while these facilities provide reliable and pervasive access to the data and data products, users typically must download the data of interest and process them using local resources. Consequently, transforming these data and data products into insights requires local access to powerful computing, storage, and networking resources. On the other hand, the NSF Advanced Cyberinfrastructure (ACI) is playing an increasingly important role as an open platform for computational and data-enabled science and engineering and can provide the necessary capabilities to allow a broad user community to effectively process the data in large facilities. However, despite clearly complementing each other, large scientific facilities and NSF ACI remain largely disconnected. As a result, users are forced to actively be part of the process that moves data from large facilities to local computational resources or NSF ACI. Therefore, this data-delivery mode becomes inefficient and limits the potential utility that the data would have if processed in an automatic manner. The outcome of this research can have a significant impact on the scientific and engineering community by improving the accessibility of data and the way scientists interact with both data sources and computational infrastructures. Bringing national ACI and large scientific facilities together will democratize access to science and improve the impact of the NSF-funded infrastructure. This is especially important for small public institutions that have limited resources and do not have high bandwidth Internet connection to the Academic/Research network. The development of human resources, including the training of students, researchers and software professionals, as well as the outreach to minorities and underrepresented groups, will be an integral aspect of this effort. The project uses an open repository to disseminate research papers, prototype implementations, and associated data products to the community.The goal of this project is to explore how NSF-funded ACI, such as the Extreme Science and Engineering Discovery Environment (XSEDE), can be integrated with large facilities generally, and the Ocean Observatories Initiative (OOI) specifically, in an automated manner to support end-to-end user workflows. Specifically, we propose to enable workflows that when triggered can seamlessly orchestrate the entire data-to-discovery pipeline. This involves executing queries on the OOI cyberinfrastructure (possibly based on the occurrence of events of interest), streaming data to appropriate ACI facilities using high bandwidth interconnects (such as Internet2) in order to stage this data close to computing/analytics resources (e.g., XSEDE JetStream), and then launching the modeling and analysis processes to transform such data into insights. In this way, the project will leverage high-performance networks that typically connect these facilities to support data movement, and process this data using state-of-the-art high-performance systems.