A New Visual Analytics Toolkit for ATLAS Computing Metadata
A New Visual Analytics Toolkit for ATLAS Computing Metadata
复制标题
用于 ATLAS 计算元数据的新可视化分析工具包
DOI：
--
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
M. Titov
中科院分区：
文献类型：
--
作者：
M. Grigorieva;A. Alekseev;T. Galkin;A. Klimentov;T. Korchuganova;I. Milman;S. Padolski;V. Pilyugin;M. Titov
大型强子对撞机上的ATLAS实验有一个复杂的异构分布式计算基础设施，用于处理和分析EB级数据。在数据处理和物理分析的所有阶段都收集和存储元数据。所有元数据可分为用于准在线监测的业务元数据和用于研究相应系统在特定时期内的行为的存档元数据（即长期数据分析）。确保复杂和大型系统的稳定性和效率，例如ATLAS Computing中的系统，需要复杂的监控工具，长期监控数据分析与监控本身一样重要。归档元数据包含了十多年来积累的大量指标（硬件和软件环境描述、网络状态、应用程序参数、错误），可以通过各种机器学习（ML）算法成功地进行分类、聚类和降维处理。然而，ML数据分析尽管使用量很大，但也并非没有缺点：底层算法通常被视为“黑匣子”，因为没有有效的技术来理解其内部机制。因此，数据分析缺乏人为监督。此外，有时算法得出的结论对于真实的数据模型可能没有意义。在这项工作中，我们将演示如何将交互式数据可视化应用于扩展常规ML数据分析方法。可视化允许积极利用人类的空间思维来识别在收集的数据中发现的新趋势和模式，避免了使用工具分析工具的必要性。介绍了ATLAS计算元数据多维数据分析可视化工具包交互式可视化浏览器（InVEx）的体系结构和相应的原型。原型的Web应用部分提供了ATLAS计算作业的交互式可视化聚类，搜索计算作业的非平凡行为及其可能的原因。
The ATLAS experiment at the Large Hadron Collider has a complex heterogeneous distributed computing infrastructure, which is used to process and analyse exabytes of data. Metadata are collected and stored at all stages of data processing and physics analysis. All metadata could be divided into operational metadata to be used for the quasi on-line monitoring, and archival to study the behaviour of corresponding systems over a given period of time (i.e. long-term data analysis). Ensuring the stability and efficiency of complex and large-scale systems, such as those in the ATLAS Computing, requires sophisticated monitoring tools, and the long-term monitoring data analysis becomes as important as the monitoring itself. Archival metadata, which contains a lot of metrics (hardware and software environment descriptions, network states, application parameters, errors) accumulated for more than a decade, can be successfully processed by various machine learning (ML) algorithms for classification, clustering and dimensionality reduction. However, the ML data analysis, despite the massive use, is not without shortcomings: the underlying algorithms are usually treated as “black boxes”, as there are no effective techniques for understanding their internal mechanisms. As a result, the data analysis suffers from the lack of human supervision. Moreover, sometimes the conclusions made by algorithms may not be making sense with regard to the real data model. In this work we will demonstrate how the interactive data visualization can be applied to extend the routine ML data analysis methods. Visualization allows an active use of human spatial thinking to identify new tendencies and patterns found in the collected data, avoiding the necessity of struggling with the instrumental analytics tools. The architecture and the corresponding prototype of Interactive Visual Explorer (InVEx) - visual analytics toolkit for the multidimensional data analysis of ATLAS computing metadata will be presented. The web-application part of the prototype provides an interactive visual clusterization of ATLAS computing jobs, search for computing jobs non-trivial behaviour and its possible reasons.