Collaborative Research: SI2-SSE: WRENCH: A Simulation Workbench for Scientific Workflow Users, Developers, and Researchers

协作研究:SI2-SSE:WRENCH:面向科学工作流程用户、开发人员和研究人员的模拟工作台

基本信息

项目摘要

Many scientific breakthroughs can only be achieved by performing complex processing of vast amounts of data efficiently. In domains as crucial to our society as climate modeling, oceanography, particle physics, seismology, or computational biology (and in fact in most fields of physics, chemistry, and biology today), scientists nowadays routinely define "scientific workflows". These workflows are complex descriptions of scientific processes as data and inter-dependent computations on these data. When executed, typically with great expenses of computing, storage, and networking hardware, these workflows can produce groundbreaking results. A famous and recent example is the workflow that was used as part of the LIGO project to confirm the first detection of gravitational waves from colliding black holes. Scientific workflows are mainstays in today's science. Their efficient execution (in terms of speed, reliability, and cost) is thus crucial. This project seeks to provide a software framework, called WRENCH (Workflow Simulation Workbench), that will make it possible to simulate large-scale hypothetical scenarios quickly and accurately on a single computer, obviating the need for expensive and time-consuming trial and error experiments. WRENCH potentially enables scientists to make quick and informed choices when executing their workflows, software developers to implement more efficient software infrastructures to support workflows, and researchers to develop novel efficient algorithms to be embedded within these software infrastructures. In addition, WRENCH makes it possible to bring scientific workflow content into undergraduate and graduate computer science curricula. This is because meaningful knowledge can be gained by students using a single computer and the WRENCH software stack, making such learning possible even at institutions without access to high-end computing infrastructures, such as many non-Ph.D.-granting and minority-serving institutions. As a result, this work will contribute to producing computer science graduates better equipped to take an active role in the advancing of science. Due to its potentially transformative impact on scientific workflow usage, development, research, and education, this project promises to promote the progress of science across virtually all its fields, ultimately resulting in broad and numerous benefits to our society.Scientific workflows have become mainstream for conducting large-scale scientific research. As a result, many workflow applications and Workflow Management Systems (WMSs) have been developed as part of the cyberinfrastructure to allow scientists to execute their applications seamlessly on a range of distributed platforms. In spite of many success stories, building large-scale workflows and orchestrating their executions efficiently (in terms of performance, reliability, and cost) remains a challenge given the complexity of the workflows themselves and the complexity of the underlying execution platforms. A fundamental necessary next step is the establishment of a solid "experimental science" approach for future workflow technology development. Such an approach is useful for scientists who need to design workflows and pick execution platforms, for WMS developers who need to compare alternate design and implementation options, and for researchers who need to develop novel decision-making algorithms to be implemented as part of WMSs. The broad objective of this work is to provide foundational software, the Workflow Simulation Workbench (WRENCH), upon which to develop the above experimental science approach. Capitalizing on recent advances in distributed application and platform simulation technology, WRENCH makes it possible to (i) quickly prototype workflow, WMS implementations, and decision-making algorithms; and (ii) evaluate/compare alternative options scalably and accurately for arbitrary, and often hypothetical, experimental scenarios. This project will define a generic and foundational software architecture, that is informed by current state-of-the-art WMS designs and planned future designs. The implementation of the components in this architecture when taken together form a generic "scientific instrument" that can be used by workflow users, developers, and researchers. This scientific instrument will be instantiated for several real-world WMSs and used for a range of real-world workflow applications. In a particular case-study, it will be used with a popular WMS (Pegasus) to revisit published results and scheduling algorithms in the area of workflow planning optimizations. The objective is to demonstrate the benefit of using an experimental science approach for WMS research. Another impact of this project is that it makes it possible to include scientific workflow content pervasively in undergraduate and graduate computer science curricula, even for students without any access to computing infrastructure, by defining meaningful pedagogic activities that only require a computer and the WRENCH software stack. This educational impact will be demonstrated in the classroom in both undergraduate and graduate courses at our institutions.
许多科学突破只能通过对大量数据进行高效的复杂处理来实现。在气候建模、海洋学、粒子物理学、地震学或计算生物学等对我们的社会至关重要的领域(事实上,在今天的大多数物理、化学和生物学领域),科学家们现在经常定义“科学工作流程”。这些工作流程是对科学过程的复杂描述,即数据和基于这些数据的相互依赖的计算。在执行这些工作流时,通常需要花费大量的计算、存储和网络硬件,但这些工作流可以产生突破性的结果。最近一个著名的例子是LIGO项目的工作流程,它被用来确认首次探测到黑洞碰撞产生的引力波。科学工作流程是当今科学的支柱。因此,它们的高效执行(在速度、可靠性和成本方面)至关重要。该项目旨在提供一个名为扳手(工作流仿真工作台)的软件框架,它将使在一台计算机上快速准确地模拟大规模假设场景成为可能,从而避免了昂贵且耗时的试验和错误实验的需要。扳手可以使科学家在执行工作流程时做出快速和明智的选择,软件开发人员可以实现更有效的软件基础设施来支持工作流程,研究人员可以开发新的高效算法,嵌入到这些软件基础设施中。此外,扳手可以把科学的工作流程内容纳入本科和研究生计算机科学课程。这是因为学生可以通过一台计算机和扳手软件栈获得有意义的知识,即使在没有高端计算基础设施的机构,例如许多非博士学位的机构,也可以进行这样的学习。-资助和少数民族服务机构。因此,这项工作将有助于培养计算机科学毕业生,使他们能够更好地在科学发展中发挥积极作用。由于其对科学工作流程的使用、开发、研究和教育的潜在变革性影响,该项目有望促进几乎所有领域的科学进步,最终为我们的社会带来广泛而众多的利益。科学工作流程已经成为进行大规模科学研究的主流。因此,许多工作流应用程序和工作流管理系统(WMSs)作为网络基础设施的一部分被开发出来,使科学家能够在一系列分布式平台上无缝地执行他们的应用程序。尽管有许多成功的案例,但考虑到工作流本身的复杂性和底层执行平台的复杂性,构建大规模工作流并有效地编排它们的执行(在性能、可靠性和成本方面)仍然是一个挑战。一个基本的、必要的下一步是为未来工作流技术的发展建立一个坚实的“实验科学”方法。对于需要设计工作流和选择执行平台的科学家、需要比较备选设计和实现方案的WMS开发人员以及需要开发作为WMS一部分实现的新决策算法的研究人员来说,这种方法非常有用。这项工作的广泛目标是提供基础软件,工作流仿真工作台(扳手),在此基础上开发上述实验科学方法。利用分布式应用程序和平台仿真技术的最新进展,扳手可以(i)快速原型工作流,WMS实现和决策算法;(ii)针对任意的、通常是假设的实验场景,可扩展地、准确地评估/比较备选方案。该项目将定义一个通用的和基本的软件架构,这是由当前最先进的WMS设计和计划的未来设计提供的信息。该体系结构中组件的实现组合在一起形成了一个通用的“科学仪器”,工作流用户、开发人员和研究人员都可以使用它。该科学仪器将在几个现实世界的工作流管理系统中实例化,并用于一系列现实世界的工作流应用程序。在一个特定的案例研究中,它将与一个流行的WMS (Pegasus)一起使用,以重新审视工作流规划优化领域的已发布结果和调度算法。目的是证明在WMS研究中使用实验科学方法的好处。该项目的另一个影响是,通过定义只需要一台计算机和扳手软件栈的有意义的教学活动,它使得科学工作流内容在本科和研究生计算机科学课程中普遍存在成为可能,甚至对于没有任何计算基础设施的学生也是如此。这种教育影响将在我们机构的本科和研究生课程的课堂上得到体现。

项目成果

期刊论文数量(7)
专著数量(0)
科研奖励数量(0)
会议论文数量(0)
专利数量(0)
Developing accurate and scalable simulators of production workflow management systems with WRENCH
使用 WRENCH 开发生产工作流程管理系统的准确且可扩展的模拟器
  • DOI:
    10.1016/j.future.2020.05.030
  • 发表时间:
    2020
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Casanova, Henri;Ferreira da Silva, Rafael;Tanaka, Ryan;Pandey, Suraj;Jethwani, Gautam;Koch, William;Albrecht, Spencer;Oeth, James;Suter, Frédéric
  • 通讯作者:
    Suter, Frédéric
WRENCH: A Framework for Simulating Workflow Management Systems
WRENCH:模拟工作流管理系统的框架
Characterizing, Modeling, and Accurately Simulating Power and Energy Consumption of I/O-intensive Scientific Workflows
  • DOI:
    10.1016/j.jocs.2020.101157
  • 发表时间:
    2020-07-01
  • 期刊:
  • 影响因子:
    3.3
  • 作者:
    da Silva, Rafael Ferreira;Casanova, Henri;Suter, Frederic
  • 通讯作者:
    Suter, Frederic
Teaching parallel and distributed computing concepts in simulation with WRENCH
使用 WRENCH 进行模拟教学并行和分布式计算概念
  • DOI:
    10.1016/j.jpdc.2021.05.009
  • 发表时间:
    2021
  • 期刊:
  • 影响因子:
    3.8
  • 作者:
    Casanova, Henri;Tanaka, Ryan;Koch, William;Ferreira da Silva, Rafael
  • 通讯作者:
    Ferreira da Silva, Rafael
Bridging Concepts and Practice in eScience via Simulation-Driven Engineering
通过仿真驱动工程连接电子科学的概念和实践
{{ item.title }}
{{ item.translation_title }}
  • DOI:
    {{ item.doi }}
  • 发表时间:
    {{ item.publish_year }}
  • 期刊:
  • 影响因子:
    {{ item.factor }}
  • 作者:
    {{ item.authors }}
  • 通讯作者:
    {{ item.author }}

数据更新时间:{{ journalArticles.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ monograph.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ sciAawards.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ conferencePapers.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ patent.updateTime }}

Rafael Ferreira da Silva其他文献

A Science-Gateway Workload Archive to Study Pilot Jobs, User Activity, Bag of Tasks, Task Sub-steps, and Workflow Executions
用于研究试点作业、用户活动、任务包、任务子步骤和工作流程执行的 Science-Gateway 工作负载存档
  • DOI:
  • 发表时间:
    2012
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Rafael Ferreira da Silva;T. Glatard
  • 通讯作者:
    T. Glatard
Workflow Performance Profiles: Development and Analysis
工作流程绩效概况:开发和分析
  • DOI:
  • 发表时间:
    2016
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Dariusz Król;Rafael Ferreira da Silva;E. Deelman;V. Lynch
  • 通讯作者:
    V. Lynch
Using Simple PID Controllers to Prevent and Mitigate Faults in Scientific Workflows
使用简单的 PID 控制器预防和减轻科学工作流程中的故障
  • DOI:
  • 发表时间:
    2016
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Rafael Ferreira da Silva;Rosa Filgueira;E. Deelman;E. Pairo;I. Overton;M. Atkinson
  • 通讯作者:
    M. Atkinson
A Cleanup Algorithm for Implementing Storage Constraints in Scientific Workflow Executions
在科学工作流程执行中实现存储约束的清理算法
  • DOI:
  • 发表时间:
    2014
  • 期刊:
  • 影响因子:
    0
  • 作者:
    S. Srinivasan;G. Juve;Rafael Ferreira da Silva;K. Vahi;E. Deelman
  • 通讯作者:
    E. Deelman
SIM-SITU: A Framework for the Faithful Simulation of in situ Processing
SIM-SITU:原位处理忠实模拟的框架

Rafael Ferreira da Silva的其他文献

{{ item.title }}
{{ item.translation_title }}
  • DOI:
    {{ item.doi }}
  • 发表时间:
    {{ item.publish_year }}
  • 期刊:
  • 影响因子:
    {{ item.factor }}
  • 作者:
    {{ item.authors }}
  • 通讯作者:
    {{ item.author }}

{{ truncateString('Rafael Ferreira da Silva', 18)}}的其他基金

CCRI: Planning: Collaborative Research: Infrastructure for Enabling Systematic Development and Research of Scientific Workflow Management Systems
CCRI:规划:协作研究:支持科学工作流程管理系统系统开发和研究的基础设施
  • 批准号:
    2016619
  • 财政年份:
    2020
  • 资助金额:
    $ 24万
  • 项目类别:
    Standard Grant

相似国自然基金

Research on Quantum Field Theory without a Lagrangian Description
  • 批准号:
    24ZR1403900
  • 批准年份:
    2024
  • 资助金额:
    0.0 万元
  • 项目类别:
    省市级项目
Cell Research
  • 批准号:
    31224802
  • 批准年份:
    2012
  • 资助金额:
    24.0 万元
  • 项目类别:
    专项基金项目
Cell Research
  • 批准号:
    31024804
  • 批准年份:
    2010
  • 资助金额:
    24.0 万元
  • 项目类别:
    专项基金项目
Cell Research (细胞研究)
  • 批准号:
    30824808
  • 批准年份:
    2008
  • 资助金额:
    24.0 万元
  • 项目类别:
    专项基金项目
Research on the Rapid Growth Mechanism of KDP Crystal
  • 批准号:
    10774081
  • 批准年份:
    2007
  • 资助金额:
    45.0 万元
  • 项目类别:
    面上项目

相似海外基金

Collaborative Research: SI2-SSI: Expanding Volunteer Computing
合作研究:SI2-SSI:扩展志愿者计算
  • 批准号:
    2039142
  • 财政年份:
    2020
  • 资助金额:
    $ 24万
  • 项目类别:
    Standard Grant
SI2-SSI: Collaborative Research: Einstein Toolkit Community Integration and Data Exploration
SI2-SSI:协作研究:Einstein Toolkit 社区集成和数据探索
  • 批准号:
    2114580
  • 财政年份:
    2020
  • 资助金额:
    $ 24万
  • 项目类别:
    Continuing Grant
Collaborative Research: SI2-SSI: Expanding Volunteer Computing
合作研究:SI2-SSI:扩展志愿者计算
  • 批准号:
    2001752
  • 财政年份:
    2019
  • 资助金额:
    $ 24万
  • 项目类别:
    Standard Grant
Collaborative Research: NISC SI2-S2I2 Conceptualization of CFDSI: Model, Data, and Analysis Integration for End-to-End Support of Fluid Dynamics Discovery and Innovation
合作研究:NISC SI2-S2I2 CFDSI 概念化:模型、数据和分析集成,用于流体动力学发现和创新的端到端支持
  • 批准号:
    1743178
  • 财政年份:
    2018
  • 资助金额:
    $ 24万
  • 项目类别:
    Continuing Grant
Collaborative Research: NISC SI2-S2I2 Conceptualization of CFDSI: Model, Data, and Analysis Integration for End-to-End Support of Fluid Dynamics Discovery and Innovation
合作研究:NISC SI2-S2I2 CFDSI 概念化:模型、数据和分析集成,用于流体动力学发现和创新的端到端支持
  • 批准号:
    1743185
  • 财政年份:
    2018
  • 资助金额:
    $ 24万
  • 项目类别:
    Continuing Grant
Collaborative Research: NISC SI2-S2I2 Conceptualization of CFDSI: Model, Data, and Analysis Integration for End-to-End Support of Fluid Dynamics Discovery and Innovation
合作研究:NISC SI2-S2I2 CFDSI 概念化:模型、数据和分析集成,用于流体动力学发现和创新的端到端支持
  • 批准号:
    1743180
  • 财政年份:
    2018
  • 资助金额:
    $ 24万
  • 项目类别:
    Continuing Grant
Collaborative Research: NISC SI2-S2I2 Conceptualization of CFDSI: Model, Data, and Analysis Integration for End-to-End Support of Fluid Dynamics Discovery and Innovation
合作研究:NISC SI2-S2I2 CFDSI 概念化:模型、数据和分析集成,用于流体动力学发现和创新的端到端支持
  • 批准号:
    1743179
  • 财政年份:
    2018
  • 资助金额:
    $ 24万
  • 项目类别:
    Continuing Grant
Collaborative Research: NISC SI2-S2I2 Conceptualization of CFDSI: Model, Data, and Analysis Integration for End-to-End Support of Fluid Dynamics Discovery and Innovation
合作研究:NISC SI2-S2I2 CFDSI 概念化:模型、数据和分析集成,用于流体动力学发现和创新的端到端支持
  • 批准号:
    1743191
  • 财政年份:
    2018
  • 资助金额:
    $ 24万
  • 项目类别:
    Continuing Grant
Collaborative Research: SI2-SSI: Expanding Volunteer Computing
合作研究:SI2-SSI:扩展志愿者计算
  • 批准号:
    1664022
  • 财政年份:
    2017
  • 资助金额:
    $ 24万
  • 项目类别:
    Standard Grant
Collaborative Research: SI2-SSI: Cyberinfrastructure for Advancing Hydrologic Knowledge through Collaborative Integration of Data Science, Modeling and Analysis
合作研究:SI2-SSI:通过数据科学、建模和分析的协作集成推进水文知识的网络基础设施
  • 批准号:
    1664061
  • 财政年份:
    2017
  • 资助金额:
    $ 24万
  • 项目类别:
    Standard Grant
{{ showInfoDetail.title }}

作者:{{ showInfoDetail.author }}

知道了