Data federation strategies for ATLAS using XRootD
Data federation strategies for ATLAS using XRootD
复制标题
使用 XRootD 的 ATLAS 数据联合策略
DOI：
10.1088/1742-6596/513/4/042049
复制
发表时间：
2014
期刊：
影响因子：
--
通讯作者：
W. Yang
中科院分区：
文献类型：
--
作者：
R. Gardner;S. Campana;G. Duckeck;J. Elmsheuser;A. Hanushevsky;F. Hönig;J. Iven;F. Legger;I. Vukotic;W. Yang
在过去的一年中，ATLAS协作加速了其计划，使用基于XRootD的架构及其附带的重定向和存储集成服务来联合数据存储资源。联合的主要目标是改善最终用户的数据访问体验，同时允许更有效和智能地使用计算资源。沿着这些进步的是与现有ATLAS生产服务（PanDA及其试点服务）和数据管理服务（DQ 2和下一代Rucio）的集成。联邦的功能测试已集成到标准ATLAS和WLCG监控框架中，一组专用工具提供了有关其当前和历史使用情况的高粒度信息。我们使用一个联合拓扑，旨在从站点的本地存储向外搜索到其所在区域和全局分布的存储资源。我们描述了各种联邦访问模式的编程测试，包括通过广域网直接访问和将远程数据文件转移到本地磁盘。为了支持工作中介的决定，一个时间依赖的数据访问成本矩阵，考虑到网络性能和关键的网站性能因素。该系统的响应生产规模的物理分析工作量，无论是从个人最终用户或ATLAS分析服务，进行了讨论。
In the past year the ATLAS Collaboration accelerated its program to federate data storage resources using an architecture based on XRootD with its attendant redirection and storage integration services. The main goal of the federation is an improvement in the data access experience for the end user while allowing more efficient and intelligent use of computing resources. Along with these advances come integration with existing ATLAS production services (PanDA and its pilot services) and data management services (DQ2, and in the next generation, Rucio). Functional testing of the federation has been integrated into the standard ATLAS and WLCG monitoring frameworks and a dedicated set of tools provides high granularity information on its current and historical usage. We use a federation topology designed to search from the site's local storage outward to its region and to globally distributed storage resources. We describe programmatic testing of various federation access modes including direct access over the wide area network and staging of remote data files to local disk. To support job-brokering decisions, a time-dependent cost-of-data-access matrix is made taking into account network performance and key site performance factors. The system's response to production-scale physics analysis workloads, either from individual end-users or ATLAS analysis services, is discussed.