SI2-SSI: Pegasus: Automating Compute and Data Intensive Science

该项目解决了校内和校外网络基础设施（CI）提供的能力与研究人员有效利用这些能力推进科学发现的能力之间日益扩大的差距。校园里的教师和学生很难从笔记本电脑上无法存储或无法用Excel电子表格处理的数据中提取知识，他们发现很难有效地管理他们的计算。该项目维持并增强了Pegasus工作流管理系统，该系统使科学家能够在不同的分布式计算资源上协调和运行数据和计算密集型计算。增强功能主要集中在Pegasus提供的自动化功能上，以支持处理大型数据集的工作流，以及Pegasus的可用性，这降低了采用Pegasus的障碍。这项工作扩大了Pegasus为研究人员提供的先进能力的范围，从引力波物理学到生物信息学，从地球科学到材料科学，范围更广。在超过15年的时间里，Pegasus工作流管理系统一直在设计、实施和支持提供抽象，使科学家能够专注于构建他们的计算，而不必担心目标CI的细节。为了支持这些工作流抽象，Pegasus提供了自动化功能，可以无缝地将工作流映射到目标资源上，从而节省了科学家管理数据流、作业调度、故障恢复和应用程序适应的开销。自动化使服务的交付能够考虑诸如解决方案的时间等标准，以及考虑资源的有效使用、管理任务的吞吐量和数据传输请求。这些抽象的力量在2015年得到了证明，当时一项国际合作利用Pegasus来利用各种资源，并管理计算和数据密集型工作流程，证实了爱因斯坦相对论所预测的引力波的存在。在天文学、生物信息学、气候建模、地震科学、重力和材料科学等不同科学领域的工作经验，为科学工作流程的进一步自动化提供了机会。该项目通过以下领域的创新来解决这些机会：自动化方法，包括在工作流执行之前和期间提供资源，数据感知作业调度算法，以及高吞吐量环境中的数据共享机制。为了支持更广泛的“长尾”科学家群体，我们致力于可用性改进以及扩展、教育和培训活动。提议的工作包括实施和评估先进的框架、算法和方法，以增强自动化的力量，支持数据密集型科学。这些增强功能作为与Pegasus集成的可靠软件工具提供，以便可以在实际应用程序和计算环境中进行评估。数据感知的重点是在高吞吐量和高性能环境中执行的新类型的应用程序。从引力波物理学到生物信息学，从地球科学到材料科学，各个学科的研究人员都采用了Pegasus。它提供并增强了对OSG和XSEDE等国家CI的访问，作为这项工作的一部分，它将部署在变色龙和Jetstream中，为NSF的CI投资提供更广泛的访问。通过可用性改进，与CI和社区平台提供商（如HubZero和Cyverse）的合作，结合教育、培训和教程活动，该项目扩大了利用自动化进行工作的研究人员的范围。与网关研究所的合作确保了Pegasus接口适合于科学网关内的垂直集成，并无缝地支持新的科学社区。

This project addresses the ever-growing gap between the capabilities offered by on-campus and off-campus cyberinfrastructures (CI) and the ability of researchers to effectively harness these capabilities to advance scientific discovery. Faculty and students on campuses struggle to extract knowledge from data that does not fit on their laptops or cannot be processed by an Excel spreadsheet and they find it difficult to efficiently manage their computations. The project sustains and enhances the Pegasus Workflow Management System, which enables scientist to orchestrate and run data- and compute-intensive computations on diverse distributed computational resources. Enhancements focus on the automation capabilities provided by Pegasus to support workflows handling large data sets, as well as usability of Pegasus that lowers the barrier of its adoption. This effort expands the reach of the advanced capabilities provided by Pegasus to researchers from a broader spectrum of disciplines that range from gravitational-wave physics to bioinformatics, and from earth science to material science.For more than 15 years the Pegasus Workflow Management System has been designed, implemented and supported to provide abstractions that enable scientists to focus on structuring their computations without worrying about the details of the target CI. To support these workflow abstractions Pegasus provides automation capabilities that seamlessly map workflows onto target resources, sparing scientists the overhead of managing the data flow, job scheduling, fault recovery and adaptation of their applications. Automation enables the delivery of services that consider criteria such as time-to-solution, as well as takes into account efficient use of resources, managing the throughput of tasks, and data transfer requests. The power of these abstractions was demonstrated in 2015 when Pegasus was used by an international collaboration to harness a diverse set of resources and to manage compute- and data- intensive workflows that confirmed the existence of gravitational waves, as predicted by Einstein's theory of relativity. Experience from working with diverse scientific domains - astronomy, bioinformatics, climate modeling, earthquake science, gravitational and material science - uncover opportunities for further automation of scientific workflows. This project addresses these opportunities through innovation in the following areas: automation methods to include resource provisioning ahead of and during workflow execution, data-aware job scheduling algorithms, and data sharing mechanisms in high-throughput environments. To support a broader group of "long-tail" scientists, effort is devoted to usability improvements as well as outreach, education, and training activities. The proposed work includes the implementation and evaluation of advanced frameworks, algorithms, and methods that enhance the power of automation in support of data-intensive science. These enhancements are delivers as dependable software tools integrated with Pegasus so that they can be evaluated in the context of real-life applications and computing environments. The data-aware focus targets new classes of applications executing in high-throughput and high-performance environments. Pegasus has been adopted by researchers from a broad spectrum of disciplines that range from gravitational-wave physics to bioinformatics, and from earth science to material science. It provides and enhances access to national CI such as OSG and XSEDE, and as part of this work it will be deployed within Chameleon and Jetstream to provide broader access to NSF's CI investments. Through usability improvements, engagement with CI and community platform providers such as HubZero and Cyverse, combined with educational, training, and tutorial activities, this project broadens the set of researchers that leverage automation for their work. Collaboration with the Gateways Institute assures that Pegasus interfaces are suitable for vertical integration within science gateways and seamlessly supports new scientific communities.