Monitoring of large-scale federated data storage: XRootD and beyond
Monitoring of large-scale federated data storage: XRootD and beyond
复制标题
监控大规模联合数据存储：XRootD 及其他
DOI：
10.1088/1742-6596/513/3/032004
复制
发表时间：
2014
期刊：
影响因子：
--
通讯作者：
I. Vukotic
中科院分区：
文献类型：
--
作者：
J. Andreeva;A. Beche;Sergey Belov;D. D. Arias;D. Giordano;D. Oleynik;A. Petrosyan;P. Saiz;M. Tadel;D. Tuckett;I. Vukotic
LHC实验的计算模型正逐渐从具有集中管理的数据预放置的分层数据模型转向联合存储，联合存储提供对数据文件的无缝访问，而不受其位置的影响，并由于故障转移机制而显着改善恢复。数据联合的构建和了解新的数据管理方法对用户分析的影响需要完整和详细的监控。监控功能应涵盖联合存储的所有组件的状态，测量数据流量和数据访问性能，以及能够检测任何类型的效率低下，并为资源优化和有效的数据分发策略提供提示。对收集的监控数据进行数据挖掘，可以深入了解新的使用模式。在WLCG上下文中，目前有几个联合基于XRootD技术。本文将重点关注在实验仪表板监控框架中实现的ATLAS和CMS XRootD联邦的监控。这两个联合会都由数十个站点组成，可供数百名客户访问，并且规模不断扩大。这些系统产生的监测流的处理必须得到很好的优化，以实现所需的性能。此外，本文还证明了XRootD监控架构足够通用，可以轻松地适用于其他技术，如HTTP/WebDAV动态联合。
The computing models of the LHC experiments are gradually moving from hierarchical data models with centrally managed data pre-placement towards federated storage which provides seamless access to data files independently of their location and dramatically improve recovery due to fail-over mechanisms. Construction of the data federations and understanding the impact of the new approach to data management on user analysis requires complete and detailed monitoring. Monitoring functionality should cover the status of all components of the federated storage, measuring data traffic and data access performance, as well as being able to detect any kind of inefficiencies and to provide hints for resource optimization and effective data distribution policy. Data mining of the collected monitoring data provides a deep insight into new usage patterns. In the WLCG context, there are several federations currently based on the XRootD technology. This paper will focus on monitoring for the ATLAS and CMS XRootD federations implemented in the Experiment Dashboard monitoring framework. Both federations consist of many dozens of sites accessed by many hundreds of clients and they continue to grow in size. Handling of the monitoring flow generated by these systems has to be well optimized in order to achieve the required performance. Furthermore, this paper demonstrates the XRootD monitoring architecture is sufficiently generic to be easily adapted for other technologies, such as HTTP/WebDAV dynamic federations.