Container-based bioinformatics with Pachyderm
Container-based bioinformatics with Pachyderm
复制标题
DOI：
10.1093/bioinformatics/bty699
复制
发表时间：
2019-03-01
期刊：
影响因子：
5.8
通讯作者：
Spjuth, Ola
中科院分区：
文献类型：
--
作者：
Novella, Jon Ander;Emami Khoonsari, Payam;Spjuth, Ola
计算生物学家面临着许多与数据大小相关的挑战，他们需要管理复杂的分析，通常包括多个阶段和多个工具，所有这些都必须部署到现代基础设施中。为了应对这些挑战并保持结果的可重复性，研究人员需要（i）在任何计算环境中运行处理阶段的可靠方法，（ii）一个明确定义的方法来协调这些处理阶段，（iii）一个数据管理层，在数据通过处理管道时跟踪数据。源工作流系统和数据管理框架，通过在容器生态系统的项目之上创建数据管道和数据版本化层来满足这些需求，并将Kubernetes作为容器编排的骨干。我们适应厚皮动物，并在生物信息学中证明其有吸引力的特性。创建了一个Helm Chart，以便研究人员可以在多种情况下使用Pachyderm。Pachyderm文件系统被扩展为支持块存储。创建了一个包装器，用于在与云无关的虚拟基础架构上启动Pachyderm。通过大型代谢组学工作流程说明了Pachyderm的好处，表明Pachyderm能够实现高效和可持续的数据科学工作流程，同时保持可重复性和可扩展性。
Motivation Computational biologists face many challenges related to data size, and they need to manage complicated analyses often including multiple stages and multiple tools, all of which must be deployed to modern infrastructures. To address these challenges and maintain reproducibility of results, researchers need (i) a reliable way to run processing stages in any computational environment, (ii) a well-defined way to orchestrate those processing stages and (iii) a data management layer that tracks data as it moves through the processing pipeline.Results Pachyderm is an open-source workflow system and data management framework that fulfils these needs by creating a data pipelining and data versioning layer on top of projects from the container ecosystem, having Kubernetes as the backbone for container orchestration. We adapted Pachyderm and demonstrated its attractive properties in bioinformatics. A Helm Chart was created so that researchers can use Pachyderm in multiple scenarios. The Pachyderm File System was extended to support block storage. A wrapper for initiating Pachyderm on cloud-agnostic virtual infrastructures was created. The benefits of Pachyderm are illustrated via a large metabolomics workflow, demonstrating that Pachyderm enables efficient and sustainable data science workflows while maintaining reproducibility and scalability.