CWL-Airflow: a lightweight pipeline manager supporting Common Workflow Language
CWL-Airflow: a lightweight pipeline manager supporting Common Workflow Language
复制标题
DOI：
10.1093/gigascience/giz084
复制
发表时间：
2019-07-01
期刊：
影响因子：
9.2
通讯作者：
Barski, Artem
中科院分区：
文献类型：
--
作者：
Kotliar, Michael;Kartashov, Andrey V.;Barski, Artem
背景资料：研究数据和计算分析量的大规模增长导致生物医学计算研究中管道管理器的使用增加。然而，超过100个这样的管理器中的每一个都使用自己的方式来描述管道，导致难以将工作流移植到不同的环境中，因此计算研究的再现性较差。出于这个原因，通用工作流语言（CWL）最近被引入作为平台无关的工作流描述的规范，并开始将现有的管道和工作流管理器过渡到CWL。调查结果：在这里，我们提出了CWL-Airflow，这是一个将CWL支持添加到Apache Airflow管道管理器的包。CWL-Airflow使用CWL 1.0版规范，可以在独立的MacOS/Linux服务器、集群或各种云平台上运行工作流。提供了用于处理染色质免疫沉淀测序数据的示例CWL流水线。结论：CWL-Airflow将为用户提供一个成熟的管道管理器的功能，并能够在Airflow可以运行的任何地方执行CWL工作流程-从笔记本电脑到集群或云环境。CWL-Airflow在Apache许可证2.0版（Apache-2.0）下可用，可以从https：//barski-lab.github.io/cwl-airflow，https://scicrunch.org/resolver/RRID：SCR 017196下载。
Background: Massive growth in the amount of research data and computational analysis has led to increased use of pipeline managers in biomedical computational research. However, each of the > 100 such managers uses its own way to describe pipelines, leading to difficulty porting workflows to different environments and therefore poor reproducibility of computational studies. For this reason, the Common Workflow Language (CWL) was recently introduced as a specification for platform-independent workflow description, and work began to transition existing pipelines and workflow managers to CWL. Findings: Herein, we present CWL-Airflow, a package that adds support for CWL to the Apache Airflow pipeline manager. CWL-Airflow uses CWL version 1.0 specification and can run workflows on stand-alone MacOS/Linux servers, on clusters, or on a variety of cloud platforms. A sample CWL pipeline for processing of chromatin immunoprecipitation sequencing data is provided. Conclusions: CWL-Airflow will provide users with the features of a fully fledged pipeline manager and the ability to execute CWL workflows anywhere Airflow can run-from a laptop to a cluster or cloud environment. CWL-Airflow is available under Apache License, version 2.0 (Apache-2.0), and can be downloaded from https://barski-lab.github.io/cwl-airflow, https://scicrunch.org/resolver/RRID: SCR 017196.