Interoperable and scalable data analysis with microservices: Applications in Metabolomics
Interoperable and scalable data analysis with microservices: Applications in Metabolomics
复制标题
通过微服务进行可互操作和可扩展的数据分析：代谢组学中的应用
DOI：
10.1101/213603
复制
发表时间：
2017
期刊：
影响因子：
--
通讯作者：
Emami Khoonsari P
中科院分区：
文献类型：
--
作者：
Emami Khoonsari P
开发一个强大而高性能的数据分析工作流程，集成所有必要的组件，同时仍然能够在多个计算节点上扩展，这是一项具有挑战性的任务。我们介绍了一种通用的方法，基于微服务架构，其中软件工具封装为Docker容器，可以连接到科学的工作流和执行使用Kubernetes容器orchestrator.ResultsWe开发了一个虚拟研究环境（VRE），它有利于快速集成新的工具和开发可扩展的和可互操作的工作流进行代谢组学数据分析。该环境可以在云资源和台式计算机上按需启动。对用户的IT专业知识要求保持在最低限度，任何新手用户都可以毫不费力地重复使用工作流程。我们验证了我们的方法在代谢组学领域的两个质谱，一个核磁共振光谱和一个通量组学研究。我们表明，该方法的规模动态增加计算资源的可用性。我们证明，该方法有利于互操作性使用的主要软件套件的集成，从而在一个交钥匙的工作流程，包括所有步骤的质谱为基础的代谢组学，包括预处理，统计和识别。微服务是一种通用的方法，可以服务于任何科学学科，并为新型的大规模综合科学开辟了道路。可用性和实施PhenoMeNal联盟维护了一个门户网站（portal.phenomenal-h2020.eu），为启动虚拟研究环境提供了GUI。GitHub资源库https://github.com/phnmnl/托管所有项目的源代码。补充信息补充数据可在Bioinformaticsonline获得。
MotivationDeveloping a robust and performant data analysis workflow that integrates all necessary components whilst still being able to scale over multiple compute nodes is a challenging task. We introduce a generic method based on the microservice architecture, where software tools are encapsulated as Docker containers that can be connected into scientific workflows and executed using the Kubernetes container orchestrator.ResultsWe developed a Virtual Research Environment (VRE) which facilitates rapid integration of new tools and developing scalable and interoperable workflows for performing metabolomics data analysis. The environment can be launched on-demand on cloud resources and desktop computers. IT-expertise requirements on the user side are kept to a minimum, and workflows can be re-used effortlessly by any novice user. We validate our method in the field of metabolomics on two mass spectrometry, one nuclear magnetic resonance spectroscopy and one fluxomics study. We showed that the method scales dynamically with increasing availability of computational resources. We demonstrated that the method facilitates interoperability using integration of the major software suites resulting in a turn-key workflow encompassing all steps for mass-spectrometry-based metabolomics including preprocessing, statistics and identification. Microservices is a generic methodology that can serve any scientific discipline and opens up for new types of large-scale integrative science.Availability and implementationThe PhenoMeNal consortium maintains a web portal (https://portal.phenomenal-h2020.eu) providing a GUI for launching the Virtual Research Environment. The GitHub repository https://github.com/phnmnl/ hosts the source code of all projects.Supplementary informationSupplementary data are available atBioinformaticsonline.