Workflows in AiiDA: Engineering a high-throughput, event-based engine for robust and modular computational workflows
Workflows in AiiDA: Engineering a high-throughput, event-based engine for robust and modular computational workflows
复制标题
DOI：
10.1016/j.commatsci.2020.110086
复制
发表时间：
2021-02-01
影响因子：
3.3
通讯作者：
Pizzi, Giovanni
中科院分区：
文献类型：
--
作者：
Uhrin, Martin;Huber, Sebastiaan P.;Pizzi, Giovanni
在过去的二十年中，计算科学领域发生了巨大的转变，将高通量计算和大数据分析纳入科学发现过程的基本支柱。这就需要开发工具和技术来处理大量数据的生成、存储和处理。在这项工作中，我们深入研究了为AiiDA提供动力的工作流引擎，AiiDA是一种被广泛采用、高度灵活且有数据库支持的信息学基础设施，强调数据的可重复性。我们详细阐述了许多基于几个重要目标而做出的设计选择：能够从在个人笔记本电脑上运行扩展到高性能超级计算机，管理运行时间从几分之一秒到数周不等的作业，并能同时扩展到数千个作业，同时最大限度地提高稳健性。简而言之，AiiDA旨在成为高通量计算科学的万能工具。除了架构，我们还概述了重要的应用程序编程接口（API）设计选择，这些选择在给予工作流编写者很大自由度的同时，引导他们编写稳健且模块化的工作流，最终使他们能够将自己的科学知识编码，造福更广泛的科学界。
Over the last two decades, the field of computational science has seen a dramatic shift towards incorporating high-throughput computation and big-data analysis as fundamental pillars of the scientific discovery process. This has necessitated the development of tools and techniques to deal with the generation, storage and processing of large amounts of data. In this work we present an in-depth look at the workflow engine powering AiiDA, a widely adopted, highly flexible and database-backed informatics infrastructure with an emphasis on data reproducibility. We detail many of the design choices that were made which were informed by several important goals: the ability to scale from running on individual laptops up to high-performance supercomputers, managing jobs with runtimes spanning from fractions of a second to weeks and scaling up to thousands of jobs concurrently, and all this while maximising robustness. In short, AiiDA aims to be a Swiss army knife for high-throughput computational science. As well as the architecture, we outline important API design choices made to give workflow writers a great deal of liberty whilst guiding them towards writing robust and modular workflows, ultimately enabling them to encode their scientific knowledge to the benefit of the wider scientific community.