FabSim3: An automation toolkit for verified simulations using high performance computing
FabSim3: An automation toolkit for verified simulations using high performance computing
复制标题
FabSim3：使用高性能计算进行验证模拟的自动化工具包
DOI：
10.1016/j.cpc.2022.108596
复制
发表时间：
2023
影响因子：
6.3
通讯作者：
Groen D
中科院分区：
文献类型：
--
作者：
Groen D
计算建模和仿真研究的一个共同特征是需要以复杂的顺序执行许多任务才能获得可用的结果。这通常涉及诸如准备输入数据、预处理、在本地或远程机器上运行模拟、后处理以及执行耦合通信、验证和/或优化等任务。此类任务可能涉及大量时间和精力的手动步骤，特别是当涉及大型集成运行的管理时。此外，随着研究工作变得更加复杂，人为错误变得更加可能和大量，从而增加了损害模拟结果可信度的风险。自动化工具可以通过减少执行这些研究任务所需的手动时间和精力、使更严格的程序易于处理以及通过减少由于手动操作数量而减少人为错误的可能性来帮助确保模拟结果的可信度。此外，通过自动化获得的效率可以帮助研究人员在其项目所施加的预算和工作量限制内进行更多研究。本文介绍了 FabSim3 的主要软件版本，并解释了我们的自动化工具包如何改进和简化研究人员和应用程序开发人员的一系列任务。 FabSim3 有助于准备、提交、执行、检索和分析仿真工作流程。通过提供适当的抽象级别，FabSim3 降低了设置和管理大规模仿真场景的复杂性，同时仍然提供对底层的透明访问以进行有效调试。该工具还有助于各种不同超级计算环境的作业提交和管理（包括文件和环境的暂存和管理）。尽管 FabSim3 本身与应用程序无关，但它支持可证明可扩展的插件系统，用户可以在其中为自己的应用程序域自动执行模拟和分析工作流程。为了强调这一点，我们简要描述了这些插件的选择，并演示了该工具包在处理大型集成工作流程方面的效率。 程序摘要程序标题：FabSim3CPC 库程序文件链接：https://doi.org/10.17632/6nfrwy7ptj.1许可规定：BSD 3-clause编程语言：Python 3 问题性质：许多方面对于获得可重复且稳健的模拟结果至关重要。例如，我们需要整理所有输入和输出以供以后审查，在轻微扰动的情况下审查模型行为，量化输入数据和已知参数的关键不确定性的传播，并分析我们无法准确说明的任何参数的敏感性。解决方案方法：FabSim3 使用一系列方法来提供自动化。这些主要包括：(i) SSH + Fabric2，以实现 SSH 命令的远程执行，(ii) 主要使用 Python 字典对象的内部参数状态空间，可以通过机器插件和用户特定的修改进行自定义，(iii) Python 模板，以快速将状态空间变量插入到超级计算脚本中，(iv) 多处理和/或 QCG-PilotJob，以实现作业数组的高效提交和执行，以及 (v) 一个灵活可安装和可修改的 Python3 插件系统，它允许用户无需修改核心代码库即可创建和定制特定于应用程序的功能。除了书面代码之外，FabSim3 还依赖于一组用户约定来保持关注点分离（特别是在机器、用户和应用程序特定设置之间）。其他注释包括……
A common feature of computational modelling and simulation research is the need to perform many tasks in complex sequences to achieve a usable result. This will typically involve tasks such as preparing input data, pre-processing, running simulations on a local or remote machine, post-processing, and performing coupling communications, validations and/or optimisations. Tasks like these can involve manual steps which are time and effort intensive, especially when it involves the management of large ensemble runs. Additionally, human errors become more likely and numerous as the research work becomes more complex, increasing the risk of damaging the credibility of simulation results. Automation tools can help ensure the credibility of simulation results by reducing the manual time and effort required to perform these research tasks, by making more rigorous procedures tractable, and by reducing the probability of human error due to a reduced number of manual actions. In addition, efficiency gained through automation can help researchers to perform more research within the budget and effort constraints imposed by their projects.This paper presents the main software release of FabSim3, and explains how our automation toolkit can improve and simplify a range of tasks for researchers and application developers. FabSim3 helps to prepare, submit, execute, retrieve, and analyze simulation workflows. By providing a suitable level of abstraction, FabSim3 reduces the complexity of setting up and managing a large-scale simulation scenario, while still providing transparent access to the underlying layers for effective debugging. The tool also facilitates job submission and management (including staging and curation of files and environments) for a range of different supercomputing environments. Although FabSim3 itself is application-agnostic, it supports a provably extensible plugin system where users automate simulation and analysis workflows for their own application domains. To highlight this, we briefly describe a selection of these plugins and we demonstrate the efficiency of the toolkit in handling large ensemble workflows.Program summaryProgram Title:FabSim3CPC Library link to program files:https://doi.org/10.17632/6nfrwy7ptj.1Licensing provisions:BSD 3-clauseProgramming language:Python 3Nature of problem:Many aspects are crucial for obtaining reproducible and robust simulation results. For instance, we need to curate all the inputs and outputs for later scrutiny, scrutinize the model behaviour under slightly perturbed circumstances, quantify the propagation of key uncertainties from input data and known parameters and analyze the sensitivity for any parameters for which the exact specification eludes us.Solution method:FabSim3 uses a range of methods to provide automation. These primarily include: (i) SSH + Fabric2 to enable remote execution of SSH commands, (ii) an internal parameter state space using primarily Python dict objects that can be customized with machine- plugin- and user-specific modifications, (iii) Python templating to quickly enable the insertion of state space variables into supercomputing scripts, (iv) multiprocessing and/or QCG-PilotJob to enable efficient submission and execution of job arrays and (v) a system of flexibly installable and modifiable Python3 plugins which allows users to create and customize application-specific functionalities without modifying the core code base. In addition to the written code, FabSim3 also relies on a set of user conventions to maintain a separation of concerns (particularly between machine-, user- and application-specific settings).Additional comments including…
DOI：
10.1007/978-1-4614-7535-4
发表时间：
2013-08
期刊：
--
影响因子：
--
作者：
A. Bouguettaya;Quan Z. Sheng;F. Daniel
通讯作者：
A. Bouguettaya;Quan Z. Sheng;F. Daniel
影响因子：
5.5
作者：
Andrew Potterton;Fouad S. Husseini;M. Southey;M. Bodkin;Alexander Heifetz;P. Coveney;A. Townsend-Nicholson
通讯作者：
Andrew Potterton;Fouad S. Husseini;M. Southey;M. Bodkin;Alexander Heifetz;P. Coveney;A. Townsend-Nicholson
2014年计算科学国际会议
DOI：
--
发表时间：
2014
期刊：
影响因子：
--
作者：
D. Abramson;M. Lees;V. Krzhizhanovskaya;J. Dongarra;P. Sloot
通讯作者：
P. Sloot