EAGER: Online Processing of Data in Large Facilities using National Advanced CyberInfrastructure

EAGER：使用国家先进网络基础设施在线处理大型设施中的数据

• 批准号: 1745246
• 负责人: Ivan Rodero
• 金额: $ 29.24万
• 依托单位: Rutgers University New Brunswick
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-09-01 至 2020-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1745246&HistoricalAwards=false
• 关键词: EAGER; Online; Processing; Data; Large

Open, large-scale scientific facilities are an essential part of science and engineering enterprise. These facilities provide shared-use infrastructure, instrumentation, and data products that are openly accessible to a broad community of researchers and/or educators. Current facilities provide increasing volumes of data and data products that have the potential to deliver new insights in a wide range of science and engineering domains. However, while these facilities provide reliable and pervasive access to the data and data products, users typically must download the data of interest and process them using local resources. Consequently, transforming these data and data products into insights requires local access to powerful computing, storage, and networking resources. On the other hand, the NSF Advanced Cyberinfrastructure (ACI) is playing an increasingly important role as an open platform for computational and data-enabled science and engineering and can provide the necessary capabilities to allow a broad user community to effectively process the data in large facilities. However, despite clearly complementing each other, large scientific facilities and NSF ACI remain largely disconnected. As a result, users are forced to actively be part of the process that moves data from large facilities to local computational resources or NSF ACI. Therefore, this data-delivery mode becomes inefficient and limits the potential utility that the data would have if processed in an automatic manner. The outcome of this research can have a significant impact on the scientific and engineering community by improving the accessibility of data and the way scientists interact with both data sources and computational infrastructures. Bringing national ACI and large scientific facilities together will democratize access to science and improve the impact of the NSF-funded infrastructure. This is especially important for small public institutions that have limited resources and do not have high bandwidth Internet connection to the Academic/Research network. The development of human resources, including the training of students, researchers and software professionals, as well as the outreach to minorities and underrepresented groups, will be an integral aspect of this effort. The project uses an open repository to disseminate research papers, prototype implementations, and associated data products to the community.The goal of this project is to explore how NSF-funded ACI, such as the Extreme Science and Engineering Discovery Environment (XSEDE), can be integrated with large facilities generally, and the Ocean Observatories Initiative (OOI) specifically, in an automated manner to support end-to-end user workflows. Specifically, we propose to enable workflows that when triggered can seamlessly orchestrate the entire data-to-discovery pipeline. This involves executing queries on the OOI cyberinfrastructure (possibly based on the occurrence of events of interest), streaming data to appropriate ACI facilities using high bandwidth interconnects (such as Internet2) in order to stage this data close to computing/analytics resources (e.g., XSEDE JetStream), and then launching the modeling and analysis processes to transform such data into insights. In this way, the project will leverage high-performance networks that typically connect these facilities to support data movement, and process this data using state-of-the-art high-performance systems.

开放式的大型科学设施是科学和工程事业的重要组成部分。这些设施提供共享使用的基础设施，仪器和数据产品，可供广泛的研究人员和/或教育工作者社区开放访问。目前的设施提供了越来越多的数据和数据产品，这些数据和产品有可能在广泛的科学和工程领域提供新的见解。然而，虽然这些设施提供了对数据和数据产品的可靠和普遍的访问，但用户通常必须下载感兴趣的数据并使用本地资源处理它们。因此，将这些数据和数据产品转化为洞察力需要本地访问强大的计算、存储和网络资源。另一方面，NSF高级网络基础设施（ACI）作为计算和数据支持的科学和工程的开放平台，正在发挥越来越重要的作用，并可以提供必要的能力，使广泛的用户社区能够有效地处理大型设施中的数据。然而，尽管大型科学设施和NSF ACI之间存在明显的互补性，但它们在很大程度上仍然是脱节的。因此，用户被迫积极参与将数据从大型设施移动到本地计算资源或NSF ACI的过程。因此，这种数据传递模式变得低效，并且限制了如果以自动方式处理数据将具有的潜在效用。这项研究的成果可以通过改善数据的可访问性以及科学家与数据源和计算基础设施的交互方式，对科学和工程界产生重大影响。将国家ACI和大型科学设施结合在一起将使科学的获取民主化，并改善NSF资助的基础设施的影响。这对于资源有限且没有高带宽互联网连接到学术/研究网络的小型公共机构尤为重要。开发人力资源，包括培训学生、研究人员和软件专业人员，以及与少数群体和代表性不足的群体开展外联活动，将是这一努力的一个组成部分。该项目使用一个开放的知识库向社区传播研究论文、原型实现和相关数据产品，其目标是探索如何将NSF资助的ACI（如极限科学与工程发现环境（XSEDE））与大型设施（特别是海洋观测站倡议（OOI））以自动化的方式集成，以支持端到端用户工作流程。具体来说，我们建议启用工作流，当触发时可以无缝地编排整个数据到发现管道。这涉及在OOI网络基础设施上执行查询（可能基于感兴趣的事件的发生），使用高带宽互连（诸如因特网2）将数据流传输到适当的ACI设施，以便将该数据放置在计算/分析资源附近（例如，XSEDE JetStream），然后启动建模和分析流程，将这些数据转化为见解。通过这种方式，该项目将利用通常连接这些设施的高性能网络来支持数据移动，并使用最先进的高性能系统处理这些数据。

项目成果

A Distributed Multi-Sensor Machine Learning Approach to Earthquake Early Warning

• DOI: 10.1609/aaai.v34i01.5376
• 发表时间: 2020-02
• 作者: Kevin Fauvel;Daniel Balouek-Thomert;D. Melgar;Pedro Silva;Anthony Simonet;Gabriel Antoniu;Alexandru Costan;Véronique Masson;M. Parashar;I. Rodero;A. Termier

Runtime Management of Data Quality for Scientific Observatories Using Edge and In-Transit Resources

使用边缘和传输中资源对科学观测站的数据质量进行运行时管理

• DOI: 10.1109/sbac-pad.2018.00053
• 发表时间: 2018
• 期刊: 2018 30th International Symposium on Computer Architecture and High Performance Computing
• 作者: Zamani, Ali Reza;Balouek-Thomert, Daniel;Villalobos, J. J.;Rodero, Ivan;Parashar, Manish
• 通讯作者: Parashar, Manish

Harnessing the Computing Continuum for Urgent Science

• DOI: 10.1145/3439602.3439618
• 发表时间: 2020-11
• 期刊: ACM SIGMETRICS Performance Evaluation Review
• 作者: Daniel Balouek-Thomert;I. Rodero;M. Parashar

Exploring the Potential of Elastic Computing Clusters in Geo-Distributed Data Centers with Fast Fabric Interconnection

通过快速结构互连探索地理分布式数据中心中弹性计算集群的潜力

• DOI: 10.1109/hpcc/smartcity/dss.2019.00135
• 发表时间: 2019
• 期刊: 2019 IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference on Data Science and Systems (HPCC/SmartCity/DSS
• 作者: Chen, Shouwei;Wang, Wensheng;Rodero, Ivan
• 通讯作者: Rodero, Ivan

Optimizing Performance and Computing Resource Management of In-memory Big Data Analytics with Disaggregated Persistent Memory

使用分解的持久内存优化内存大数据分析的性能和计算资源管理

• DOI: 10.1109/ccgrid.2019.00012
• 发表时间: 2019
• 期刊: Cloud and Grid Computing (CCGRID
• 作者: Chen, Shouwei;Wang, Wensheng;Wu, Xueyang;Fan, Zhen;Huang, Kunwu;Zhuang, Peiyu;Li, Yue;Rodero, Ivan;Parashar, Manish;Weng, Dennis
• 通讯作者: Weng, Dennis