SI2-SSI: Pegasus: Automating Compute and Data Intensive Science

SI2-SSI：Pegasus：自动化计算和数据密集型科学

• 批准号: 1664162
• 负责人: Ewa Deelman
• 金额: $ 250万
• 依托单位: University of Southern California
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-05-15 至 2023-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1664162&HistoricalAwards=false
• 关键词: SI2; SSI; Pegasus; Automating; Compute

This project addresses the ever-growing gap between the capabilities offered by on-campus and off-campus cyberinfrastructures (CI) and the ability of researchers to effectively harness these capabilities to advance scientific discovery. Faculty and students on campuses struggle to extract knowledge from data that does not fit on their laptops or cannot be processed by an Excel spreadsheet and they find it difficult to efficiently manage their computations. The project sustains and enhances the Pegasus Workflow Management System, which enables scientist to orchestrate and run data- and compute-intensive computations on diverse distributed computational resources. Enhancements focus on the automation capabilities provided by Pegasus to support workflows handling large data sets, as well as usability of Pegasus that lowers the barrier of its adoption. This effort expands the reach of the advanced capabilities provided by Pegasus to researchers from a broader spectrum of disciplines that range from gravitational-wave physics to bioinformatics, and from earth science to material science.For more than 15 years the Pegasus Workflow Management System has been designed, implemented and supported to provide abstractions that enable scientists to focus on structuring their computations without worrying about the details of the target CI. To support these workflow abstractions Pegasus provides automation capabilities that seamlessly map workflows onto target resources, sparing scientists the overhead of managing the data flow, job scheduling, fault recovery and adaptation of their applications. Automation enables the delivery of services that consider criteria such as time-to-solution, as well as takes into account efficient use of resources, managing the throughput of tasks, and data transfer requests. The power of these abstractions was demonstrated in 2015 when Pegasus was used by an international collaboration to harness a diverse set of resources and to manage compute- and data- intensive workflows that confirmed the existence of gravitational waves, as predicted by Einstein's theory of relativity. Experience from working with diverse scientific domains - astronomy, bioinformatics, climate modeling, earthquake science, gravitational and material science - uncover opportunities for further automation of scientific workflows. This project addresses these opportunities through innovation in the following areas: automation methods to include resource provisioning ahead of and during workflow execution, data-aware job scheduling algorithms, and data sharing mechanisms in high-throughput environments. To support a broader group of "long-tail" scientists, effort is devoted to usability improvements as well as outreach, education, and training activities. The proposed work includes the implementation and evaluation of advanced frameworks, algorithms, and methods that enhance the power of automation in support of data-intensive science. These enhancements are delivers as dependable software tools integrated with Pegasus so that they can be evaluated in the context of real-life applications and computing environments. The data-aware focus targets new classes of applications executing in high-throughput and high-performance environments. Pegasus has been adopted by researchers from a broad spectrum of disciplines that range from gravitational-wave physics to bioinformatics, and from earth science to material science. It provides and enhances access to national CI such as OSG and XSEDE, and as part of this work it will be deployed within Chameleon and Jetstream to provide broader access to NSF's CI investments. Through usability improvements, engagement with CI and community platform providers such as HubZero and Cyverse, combined with educational, training, and tutorial activities, this project broadens the set of researchers that leverage automation for their work. Collaboration with the Gateways Institute assures that Pegasus interfaces are suitable for vertical integration within science gateways and seamlessly supports new scientific communities.

该项目解决了校内和校外网络基础设施（CI）提供的能力与研究人员有效利用这些能力来推进科学发现的能力之间不断扩大的差距。校园里的教师和学生努力从不适合他们的笔记本电脑或无法通过Excel电子表格处理的数据中提取知识，他们发现很难有效地管理他们的计算。该项目维持和增强了Pegasus工作流管理系统，使科学家能够在不同的分布式计算资源上协调和运行数据和计算密集型计算。增强功能主要集中在Pegasus提供的自动化功能，以支持处理大型数据集的工作流程，以及Pegasus的可用性，从而降低了采用的障碍。这一努力扩大了Pegasus提供的先进功能的范围，使研究人员能够从更广泛的学科，从引力波物理学到生物信息学，从地球科学到材料科学。实现并支持提供抽象，使科学家能够专注于构建他们的计算，而无需担心细节目标线人的身份为了支持这些工作流抽象，Pegasus提供了自动化功能，可以将工作流无缝映射到目标资源上，从而使科学家不必管理数据流、作业调度、故障恢复和应用程序调整。自动化使服务的交付能够考虑解决问题的时间等标准，并考虑资源的有效使用，管理任务的吞吐量和数据传输请求。这些抽象的力量在2015年得到了证明，当时Pegasus被一个国际合作组织用于利用各种资源并管理计算和数据密集型工作流程，这些工作流程证实了引力波的存在，正如爱因斯坦相对论所预测的那样。在天文学、生物信息学、气候建模、地震科学、引力和材料科学等不同科学领域的工作经验，为科学工作流程的进一步自动化提供了机会。该项目通过以下领域的创新来解决这些机会：自动化方法，包括工作流执行之前和期间的资源配置，数据感知作业调度算法以及高吞吐量环境中的数据共享机制。为了支持更广泛的“长尾”科学家群体，努力致力于可用性改进以及推广，教育和培训活动。拟议的工作包括实施和评估先进的框架、算法和方法，以增强自动化的能力，支持数据密集型科学。这些增强功能是作为与Pegasus集成的可靠软件工具提供的，因此可以在实际应用程序和计算环境中对其进行评估。数据感知的重点是在高吞吐量和高性能环境中执行的新型应用程序。Pegasus已经被来自广泛学科的研究人员采用，从引力波物理学到生物信息学，从地球科学到材料科学。它提供并增强了对OSG和XSEDE等国家CI的访问，作为这项工作的一部分，它将部署在Chameleon和Jetstream中，以提供对NSF CI投资的更广泛访问。通过可用性改进，与CI和社区平台提供商（如HubZero和Cyverse）的合作，结合教育，培训和教程活动，该项目扩大了利用自动化进行工作的研究人员的范围。与网关研究所的合作确保了Pegasus接口适合科学网关内的垂直整合，并无缝支持新的科学社区。

项目成果

FlyNet: Drones on the Horizon

FlyNet：地平线上的无人机

• DOI: 10.1109/mic.2023.3260440
• 发表时间: 2023
• 期刊: IEEE Internet Computing
• 影响因子: 3.2
• 作者: Morel, Alicia Esquivel;Qu, Chengyi;Calyam, Prasad;Wang, Cong;Thareja, Komal;Mandal, Anirban;Lyons, Eric;Zink, Michael;Papadimitriou, George;Deelman, Ewa
• 通讯作者: Deelman, Ewa

Emerging Frameworks for Advancing Scientific Workflows Research, Development, and Education

推进科学工作流程研究、开发和教育的新兴框架

• DOI: 10.1109/works54523.2021.00015
• 发表时间: 2021
• 期刊: 2021 IEEE Workshop on Workflows in Support of Large-Scale Science (WORKS
• 作者: Casanova, Henri;Deelman, Ewa;Gesing, Sandra;Hildreth, Michael;Hudson, Stephen;Koch, William;Larson, Jeffrey;McDowell, Mary Ann;Meyers, Natalie;Navarro, John-Luke
• 通讯作者: Navarro, John-Luke

Mining Literature-Based Knowledge Graph for Predicting Combination Therapeutics: A COVID-19 Use Case

• DOI: 10.1109/ickg55886.2022.00018
• 发表时间: 2022-11
• 期刊: 2022 IEEE International Conference on Knowledge Graph (ICKG)
• 作者: A. Hamed;Jakub Jończyk;Mohammad Zaiyan Alam;E. Deelman;Byung Suk Lee

The Evolution of the Pegasus Workflow Management Software

• DOI: 10.1109/mcse.2019.2919690
• 发表时间: 2019-07-01
• 期刊: COMPUTING IN SCIENCE & ENGINEERING
• 影响因子: 2.1
• 作者: Deelman, Ewa;Vahi, Karan;Livny, Miron
• 通讯作者: Livny, Miron

Performance assessment of ensembles of in situ workflows under resource constraints

• DOI: 10.1002/cpe.7111
• 发表时间: 2022-06-08
• 期刊: CONCURRENCY AND COMPUTATION-PRACTICE & EXPERIENCE
• 影响因子: 2
• 作者: Do，Tu Mai Anh;Pottier，Loic;Deelman，Ewa
• 通讯作者: Deelman，Ewa