WfCommons: A framework for enabling scientific workflow research and development
WfCommons: A framework for enabling scientific workflow research and development
复制标题
DOI：
10.1016/j.future.2021.09.043
复制
发表时间：
2021-10-13
影响因子：
7.5
通讯作者：
da Silva, Rafael Ferreira
中科院分区：
文献类型：
--
作者：
Coleman, Taina;Casanova, Henri;da Silva, Rafael Ferreira
科学工作流程是现代科学计算的基石。它们用于描述需要对大量数据进行有效且可靠的管理的复杂计算应用程序，这些数据通常在异质的分布式资源上存储/处理。工作流研究和开发社区已经采用了多种方法来对现有和新颖的工作流程算法和系统进行定量评估。特别是，一种常见的方法是模拟工作流执行。在以前的作品中，我们提出了一系列工具，这些工具已被社区采用，用于进行工作流研究。尽管他们很受欢迎，但它们仍存在一些缺点，这些缺点可阻止轻松采用，维护和与生产工作流的不断发展的结构和计算要求。在这项工作中，我们提出了WFCommons，该框架提供了用于分析工作流执行，生产合成工作流的生成器以及用于模拟工作流执行的工具的集合。我们通过将其模拟执行与真实的工作流执行进行比较，证明了生成的合成工作流的现实主义。我们还将这些结果与使用先前可用的工具集合时获得的结果进行了对比。我们发现，由我们的框架自动构建的工作流生成器不仅生成代表性的相同尺度工作流程（即，结构和任务特征分布类似于现实世界中观察到的结构和任务特征分布），而且还可以在比例更大的尺度上这样做可用的现实世界工作流程。最后，我们进行了一项案例研究，以证明框架对估计大规模工作流执行能源消耗的有用性。 （c）2021 Elsevier B.V.保留所有权利。
Scientific workflows are a cornerstone of modern scientific computing. They are used to describe complex computational applications that require efficient and robust management of large volumes of data, which are typically stored/processed on heterogeneous, distributed resources. The workflow research and development community has employed a number of methods for the quantitative evaluation of existing and novel workflow algorithms and systems. In particular, a common approach is to simulate workflow executions. In previous works, we have presented a collection of tools that have been adopted by the community for conducting workflow research. Despite their popularity, they suffer from several shortcomings that prevent easy adoption, maintenance, and consistency with the evolving structures and computational requirements of production workflows. In this work, we present WfCommons, a framework that provides a collection of tools for analyzing workflow executions, for producing generators of synthetic workflows, and for simulating workflow executions. We demonstrate the realism of the generated synthetic workflows by comparing their simulated executions to real workflow executions. We also contrast these results with results obtained when using the previously available collection of tools. We find that the workflow generators that are automatically constructed by our framework not only generate representative same-scale workflows (i.e., with structures and task characteristics distributions that resemble those observed in real-world workflows), but also do so at scales larger than that of available real-world workflows. Finally, we conduct a case study to demonstrate the usefulness of our framework for estimating the energy consumption of large-scale workflow executions. (c) 2021 Elsevier B.V. All rights reserved.