Standalone containers with ATLAS offline software
Standalone containers with ATLAS offline software
复制标题
带有 ATLAS 离线软件的独立容器
作者：
M. Vogel;M. Borodin;A. Forti;L. Heinrich
本文介绍了部署离线软件的ATLAS实验在大型强子对撞机在容器中使用的生产工作流程，如模拟和重建。为了实现这一目标，我们使用Docker和Singularity，这两种技术都是轻量级的虚拟化技术，可以将软件包封装在完整的文件系统中。通过容器部署离线版本消除了作业执行所需的运行时环境与站点上计算节点配置之间的相互依赖性。Docker或Singularity将为网格、HPC和各种机会资源提供统一的运行时环境。此外，可以用检测器的条件数据来补充发布，从而消除对计算节点处的网络连接的需要，这通常对于HPC是非常受限的。为了实现这一目标，我们构建了Docker和Singularity镜像，其中包含ATLAS软件的单个完整版本，用于在没有网络连接的运行时环境中运行探测器模拟和重建作业。与通过将每个可能的工作流的所有可能的依赖关系打包到重映像（200GB）中来生产容器的类似努力不同，我们的方法是只包含特定工作流所需的内容，并通过软件包管理器有效地管理依赖关系。这种方法导致更稳定的打包版本，其中依赖关系清晰，并且生成的映像具有更大的可移植性（16GB）。为了覆盖更广泛的工作流程，我们正在部署可用于原始数据重建的图像。由于在处理数据时访问实验条件有效载荷期间的高数据库资源消耗，这尤其具有挑战性。我们在这里描述了一个原型管道，其中图像只提供满足作业要求所需的条件有效载荷。这种按需数据库方法将使图像保持苗条，便携，并能够在没有网络连接的环境中以独立的方式支持各种工作流程。
This paper describes the deployment of the offline software of the ATLAS experiment at LHC in containers for use in production workflows such as simulation and reconstruction. To achieve this goal we are using Docker and Singularity, which are both lightweight virtualization technologies that can encapsulate software packages inside complete file systems. The deployment of offline releases via containers removes the interdependence between the runtime environment needed for job execution and the configuration of the computing nodes at the sites. Docker or Singularity would provide a uniform runtime environment for the grid, HPCs and for a variety of opportunistic resources. Additionally, releases may be supplemented with a detector’s conditions data, thus removing the need for network connectivity at computing nodes, which is normally quite restricted for HPCs. In preparation to achieve this goal, we have built Docker and Singularity images containing single full releases of ATLAS software for running detector simulation and reconstruction jobs in runtime environments without a network connection. Unlike similar efforts to produce containers by packing all possible dependencies of every possible workflow into heavy images (≈ 200GB), our approach is to include only what is needed for specific workflows and to manage dependencies efficiently via software package managers. This approach leads to more stable packaged releases where the dependencies are clear and the resulting images have more portable sizes ( 16GB). In an effort to cover a wider variety of workflows, we are deploying images that can be used in raw data reconstruction. This is particularly challenging due to the high database resource consumption during the access to the experiment’s conditions payload when processing data. We describe here a prototype pipeline in which images are provisioned only with the conditions payload necessary to satisfy the jobs’ requirements. This database-on-demand approach would keep images slim, portable and capable of supporting various workflows in a standalone fashion in environments with no network connectivity.