SAASFEE: Scalable Scientific Workflow Execution Engine
SAASFEE: Scalable Scientific Workflow Execution Engine
复制标题
DOI：
10.14778/2824032.2824094
复制
发表时间：
2015-08
期刊：
影响因子：
--
通讯作者：
M. Bux;Jörgen Brandt;Carsten Lipka;Kamal Hakimzadeh;J. Dowling;U. Leser
中科院分区：
文献类型：
--
作者：
M. Bux;Jörgen Brandt;Carsten Lipka;Kamal Hakimzadeh;J. Dowling;U. Leser
在许多科学领域，诸如传感器读数、时间序列和基因组序列等原始数据集是通过复杂的专业工具链和脚本进行分析的，这些工具和脚本以特定领域的文件格式交换中间结果。科学工作流管理系统（SWfMS）通过提供工作流规范语言、图形编辑器、容错执行引擎等来支持这些工具链的开发和执行。然而，由于对分布式计算的支持不足，许多科学工作流管理系统无法处理大型数据集。另一方面，大多数支持分布式计算的科学工作流管理系统仅允许静态的任务执行顺序。我们介绍SAASFEE，这是一个能在Hadoop YARN上运行任意复杂工作流的科学工作流管理系统。工作流是用楔形文字（Cuneiform）来指定的，这是一种侧重于并行化以及易于集成现有软件的函数式工作流语言。楔形文字工作流在Hi - WAY上执行，Hi - WAY是一个在YARN上运行工作流的高级调度器。SAASFEE的显著特点是能够执行迭代工作流、具有自适应任务调度器、可重新执行的溯源跟踪以及与选定的其他工作流系统的兼容性。在演示中，我们使用来自基因组学领域的实际工作流展示SAASFEE的所有组件。
Across many fields of science, primary data sets like sensor read-outs, time series, and genomic sequences are analyzed by complex chains of specialized tools and scripts exchanging intermediate results in domain-specific file formats. Scientific workflow management systems (SWfMSs) support the development and execution of these tool chains by providing workflow specification languages, graphical editors, fault-tolerant execution engines, etc. However, many SWfMSs are not prepared to handle large data sets because of inadequate support for distributed computing. On the other hand, most SWfMSs that do support distributed computing only allow static task execution orders. We present SAASFEE, a SWfMS which runs arbitrarily complex workflows on Hadoop YARN. Workflows are specified in Cuneiform, a functional workflow language focusing on parallelization and easy integration of existing software. Cuneiform workflows are executed on Hi-WAY, a higher-level scheduler for running workflows on YARN. Distinct features of SAASFEE are the ability to execute iterative workflows, an adaptive task scheduler, re-executable provenance traces, and compatibility to selected other workflow systems. In the demonstration, we present all components of SAASFEE using real-life workflows from the field of genomics.