Semantic-aware Workflow Construction and Analysis for Distributed Data Analytics Systems
Semantic-aware Workflow Construction and Analysis for Distributed Data Analytics Systems
复制标题
DOI：
10.1145/3307681.3325404
复制
发表时间：
2019-06
期刊：
影响因子：
--
通讯作者：
Aidi Pi;Wei Chen-;Shaoqi Wang;Xiaobo Zhou
中科院分区：
文献类型：
--
作者：
Aidi Pi;Wei Chen-;Shaoqi Wang;Xiaobo Zhou
日志记录是在分布式系统的系统工作流中记录重要事件的通用方法。现有的日志分析工具忽略了对工作流构建和分析至关重要的语义知识。此外，它们还专注于基础设施级别的分布式系统。由于日志功能的根本差异，它们在分布式数据分析系统中是无效的。提出了一种面向分布式数据分析系统的语义感知非侵入式工作流重构工具IntelLog。它能够根据目标系统生成的日志构建组件和事件之间的分层关系，而这些系统几乎不具备领域知识。利用自然语言处理，IntelLog自动提取和格式化每个日志消息中的语义信息，包括系统事件、标识符、位置信息和度量值。它通过命名约定构建一个图来表示目标系统中组件的层次关系。我们为Hadoop MapReduce、Spark和Tez实现了IntelLog。评估结果表明，IntelLog提供了具有语义的系统工作流细粒度视图。在自动检测由实际问题、错误配置和系统错误引起的异常方面，它的表现优于现有工具。用户可以查询格式化的语义知识，以了解并进一步排除系统故障。
Logging is a universal approach to recording important events in system workflows of distributed systems. Current log analysis tools ignore the semantic knowledge that is key to workflow construction and analysis. In addition, they focus on infrastructure-level distributed systems. Because of fundamental differences in log features, they are ineffective in distributed data analytics systems. This paper proposes IntelLog, a semantic-aware non-intrusive workflow reconstruction tool for distributed data analytics systems. It is capable of building hierarchical relationships between components and events from logs generated by the targeted systems with little or even no domain knowledge. Leveraging natural language processing, IntelLog automatically extracts and formats semantic information in each log message, including system events, identifiers, locality information, and metrics values. It builds a graph to represent the hierarchical relationship of components in the targeted system via nomenclature conventions. We implement IntelLog for Hadoop MapReduce, Spark and Tez. Evaluation results show that IntelLog provides a fine-grained view of the system workflows with semantics. It outperforms existing tools in automatically detecting anomalies caused by real-world problems, misconfigurations and system bugs. Users can query the formatted semantic knowledge to understand and further troubleshoot the systems.