EAGER: Online Processing of Data in Large Facilities using National Advanced CyberInfrastructure

开放式、大型科学设施是科学与工程事业的重要组成部分。这些设施提供可供广大研究人员和/或教育工作者公开访问的共享使用的基础设施、仪器和数据产品。目前的设施提供了越来越多的数据和数据产品，有可能在广泛的科学和工程领域提供新的见解。然而，虽然这些设施提供了对数据和数据产品的可靠和普遍的访问，但用户通常必须下载感兴趣的数据并使用本地资源对其进行处理。因此，将这些数据和数据产品转化为洞察力需要本地访问强大的计算、存储和网络资源。另一方面，NSF高级数字基础设施(ACI)作为计算和数据支持的科学和工程的开放平台正在发挥越来越重要的作用，并可以提供必要的能力，使广大用户社区能够在大型设施中有效地处理数据。然而，尽管大型科学设施和NSF ACI明显相辅相成，但它们在很大程度上仍然是脱节的。因此，用户被迫积极参与将数据从大型设施转移到本地计算资源或NSF ACI的过程。因此，这种数据传递模式变得低效，并限制了如果以自动方式处理数据将具有的潜在效用。这项研究的结果可以改善数据的可及性以及科学家与数据源和计算基础设施的互动方式，从而对科学界和工程界产生重大影响。将国家ACI和大型科学设施结合在一起，将使获得科学的机会民主化，并改善NSF资助的基础设施的影响。这对于资源有限且没有高带宽互联网连接到学术/研究网络的小型公共机构尤其重要。人力资源的开发，包括对学生、研究人员和软件专业人员的培训，以及与少数群体和任职人数不足的群体的接触，将是这一努力的一个组成部分。该项目使用开放式存储库向社区传播研究论文、原型实现和相关数据产品。该项目的目标是探索如何将NSF资助的ACI，如极端科学和工程发现环境(XSEDE)，以自动化的方式与大型设施集成，特别是与海洋天文台倡议(OOI)集成，以支持端到端用户工作流。具体地说，我们建议启用在触发时可以无缝编排整个数据到发现管道的工作流。这涉及在OOI网络基础设施上执行查询(可能基于感兴趣事件的发生)，使用高带宽互连(例如Internet2)将数据流传输到适当的ACI设施，以便将该数据转移到接近计算/分析资源(例如，XSEDE jetstream)的地方，然后启动建模和分析过程以将这些数据转换为洞察力。通过这种方式，该项目将利用通常连接这些设施的高性能网络来支持数据移动，并使用最先进的高性能系统处理这些数据。

Open, large-scale scientific facilities are an essential part of science and engineering enterprise. These facilities provide shared-use infrastructure, instrumentation, and data products that are openly accessible to a broad community of researchers and/or educators. Current facilities provide increasing volumes of data and data products that have the potential to deliver new insights in a wide range of science and engineering domains. However, while these facilities provide reliable and pervasive access to the data and data products, users typically must download the data of interest and process them using local resources. Consequently, transforming these data and data products into insights requires local access to powerful computing, storage, and networking resources. On the other hand, the NSF Advanced Cyberinfrastructure (ACI) is playing an increasingly important role as an open platform for computational and data-enabled science and engineering and can provide the necessary capabilities to allow a broad user community to effectively process the data in large facilities. However, despite clearly complementing each other, large scientific facilities and NSF ACI remain largely disconnected. As a result, users are forced to actively be part of the process that moves data from large facilities to local computational resources or NSF ACI. Therefore, this data-delivery mode becomes inefficient and limits the potential utility that the data would have if processed in an automatic manner. The outcome of this research can have a significant impact on the scientific and engineering community by improving the accessibility of data and the way scientists interact with both data sources and computational infrastructures. Bringing national ACI and large scientific facilities together will democratize access to science and improve the impact of the NSF-funded infrastructure. This is especially important for small public institutions that have limited resources and do not have high bandwidth Internet connection to the Academic/Research network. The development of human resources, including the training of students, researchers and software professionals, as well as the outreach to minorities and underrepresented groups, will be an integral aspect of this effort. The project uses an open repository to disseminate research papers, prototype implementations, and associated data products to the community.The goal of this project is to explore how NSF-funded ACI, such as the Extreme Science and Engineering Discovery Environment (XSEDE), can be integrated with large facilities generally, and the Ocean Observatories Initiative (OOI) specifically, in an automated manner to support end-to-end user workflows. Specifically, we propose to enable workflows that when triggered can seamlessly orchestrate the entire data-to-discovery pipeline. This involves executing queries on the OOI cyberinfrastructure (possibly based on the occurrence of events of interest), streaming data to appropriate ACI facilities using high bandwidth interconnects (such as Internet2) in order to stage this data close to computing/analytics resources (e.g., XSEDE JetStream), and then launching the modeling and analysis processes to transform such data into insights. In this way, the project will leverage high-performance networks that typically connect these facilities to support data movement, and process this data using state-of-the-art high-performance systems.