Watchdog 2.0: New developments for reusability, reproducibility, and workflow execution
Watchdog 2.0: New developments for reusability, reproducibility, and workflow execution
复制标题
DOI：
10.1093/gigascience/giaa068
复制
发表时间：
2020-06-01
期刊：
影响因子：
9.2
通讯作者：
Friedel, Caroline C.
中科院分区：
文献类型：
--
作者：
Kluge, Michael;Friedl, Marie-Sophie;Friedel, Caroline C.
背景：高通量方法的进步给生物数据分析带来了新的挑战，通常需要对大量样本应用许多相互依赖的步骤。为了应对这一挑战，开发了 Watchdog 等工作流程管理系统，以支持科学家（半）自动执行大型分析工作流程。实施：在这里，我们介绍 Watchdog 2.0，它实现了模块创建、可重用性和文档以及分析和工作流程执行的可重复性的新开发。开发内容包括用于从软件帮助页面半自动创建模块的图形用户界面、模块和工作流程的共享存储库以及标准化的模块文档格式。后者允许生成公共和用户特定模块的定制参考书。此外，工作流程执行、模块和软件版本的广泛日志记录以及对包管理器和容器虚拟化的明确支持现在确保了结果的可重复性。从日志文件生成的逐步分析协议可以例如用作手稿方法部分的草稿。最后，实现了两种新的执行模式。一种方法允许在中断或修改后恢复工作流执行，而无需重新运行不受更改影响的已成功执行的任务。第二个允许分离和重新附加到本地计算机上的工作流执行，同时任务继续在计算机集群上运行。结论：Watchdog 2.0 提供了一些新的发展，我们认为这些新的发展有益于大规模生物信息学分析，并且其他竞争的工作流管理系统并未完全涵盖这些发展。
Background: Advances in high-throughput methods have brought new challenges for biological data analysis, often requiring many interdependent steps applied to a large number of samples. To address this challenge, workflow management systems, such as Watchdog, have been developed to support scientists in the (semi-)automated execution of large analysis workflows. Implementation: Here, we present Watchdog 2.0, which implements new developments for module creation, reusability, and documentation and for reproducibility of analyses and workflow execution. Developments include a graphical user interface for semi-automatic module creation from software help pages, sharing repositories for modules and workflows, and a standardized module documentation format. The latter allows generation of a customized reference book of public and user-specific modules. Furthermore, extensive logging of workflow execution, module and software versions, and explicit support for package managers and container virtualization now ensures reproducibility of results. A step-by-step analysis protocol generated from the log file may, e.g., serve as a draft of a manuscript methods section. Finally, 2 new execution modes were implemented. One allows resuming workflow execution after interruption or modification without rerunning successfully executed tasks not affected by changes. The second one allows detaching and reattaching to workflow execution on a local computer while tasks continue running on computer clusters. Conclusions: Watchdog 2.0 provides several new developments that we believe to be of benefit for large-scale bioinformatics analysis and that are not completely covered by other competing workflow management systems.