Rucio beyond ATLAS: experiences from Belle II, CMS, DUNE, EISCAT3D, LIGO/VIRGO, SKA, XENON
Rucio beyond ATLAS: experiences from Belle II, CMS, DUNE, EISCAT3D, LIGO/VIRGO, SKA, XENON
复制标题
DOI：
10.1051/epjconf/202024511006
复制
发表时间：
2020
影响因子：
--
通讯作者：
M. Lassnig;M. Barisits;Paul Laycock;C. Serfon;E. Vaandering;K. Ellis;R. Illingworth;V. Garonne;John K. White;J. Clark;G. Fronzé;Rohini Joshi;I. Johnson;B. Bauermeister
中科院分区：
文献类型：
--
作者：
M. Lassnig;M. Barisits;Paul Laycock;C. Serfon;E. Vaandering;K. Ellis;R. Illingworth;V. Garonne;John K. White;J. Clark;G. Fronzé;Rohini Joshi;I. Johnson;B. Bauermeister
对于许多科学项目来说，数据管理是一项日益复杂的挑战。产生前所未有的数据量的数据密集型仪器的数量正在增加，其相应的工作流程也变得更加复杂。它们的存储和计算资源是异构的，分布在属于不同管理域和组织的许多地理位置。这些地点不一定与数据产生的地点一致，也不一定与数据存储、研究人员分析或存档以供安全长期存储的地点一致。为了满足这些需求，开发了数据管理系统Rucio，以允许LHC的高能物理实验ATLAS以高效和可扩展的方式管理其大量数据。但ATLAS并不孤单，几个不同的科学项目已经开始评估，采用和调整Rucio系统以满足自己的需求。随着Rucio社区的发展，引入了许多改进，添加了自定义功能，并修复了许多错误。此外，还对新的故障进行了调查，并记录了运行经验。在这篇文章中，我们收集和比较常见的成功，陷阱，并在多个不同的实验的评估工作中出现的古怪，并将它们与ATLAS的经验进行比较。这包括高能物理实验贝儿II和CMS，中微子实验DUNE，散射雷达实验EISCAT 3D，引力波观测站LIGO和VIRGO，SKA射电望远镜和暗物质搜索实验XENON。
For many scientific projects, data management is an increasingly complicated challenge. The number of data-intensive instruments generating unprecedented volumes of data is growing and their accompanying workflows are becoming more complex. Their storage and computing resources are heterogeneous and are distributed at numerous geographical locations belonging to different administrative domains and organisations. These locations do not necessarily coincide with the places where data is produced nor where data is stored, analysed by researchers, or archived for safe long-term storage. To fulfil these needs, the data management system Rucio has been developed to allow the high-energy physics experiment ATLAS at LHC to manage its large volumes of data in an efficient and scalable way. But ATLAS is not alone, and several diverse scientific projects have started evaluating, adopting, and adapting the Rucio system for their own needs. As the Rucio community has grown, many improvements have been introduced, customisations have been added, and many bugs have been fixed. Additionally, new dataflows have been investigated and operational experiences have been documented. In this article we collect and compare the common successes, pitfalls, and oddities that arose in the evaluation efforts of multiple diverse experiments, and compare them with the ATLAS experience. This includes the high-energy physics experiments Belle II and CMS, the neutrino experiment DUNE, the scattering radar experiment EISCAT3D, the gravitational wave observatories LIGO and VIRGO, the SKA radio telescope, and the dark matter search experiment XENON.