Real-Time System Log Monitoring/Analytics Framework
Real-Time System Log Monitoring/Analytics Framework
复制标题
实时系统日志监控/分析框架
DOI：
--
复制
发表时间：
2011
期刊：
影响因子：
--
通讯作者：
Raghul Gunasekaran
中科院分区：
文献类型：
--
作者：
S. Oral;D. Dillow;Byung H. Park;G. Shipman;A. Geist;Raghul Gunasekaran
分析系统日志为识别系统/应用程序异常提供了有用的见解，并有助于更好地使用系统资源。然而，对于大型系统来说，定期扫描原始日志消息是不切实际的。首先，非结构化日志消息的数量会影响可读性，其次，将日志消息与系统事件相关联是一项艰巨的任务。这些因素限制了大规模系统日志主要用于生成已知系统事件的警报，以及用于识别影响系统性能的先前未知系统事件的事后诊断。在本文中，我们描述了一个日志监控框架，使系统事件的即时分析实时。我们的基于Web的框架提供了控制台、监视器、消费者和apsched日志的真实的实时汇总视图。分析和处理日志以生成应用程序、消息类型、计算节点的个体/组以及计算平台的部分的视图。此外，从过去的应用程序运行，我们建立了一个统计配置文件的用户/应用程序的特征，相对于已知的系统事件，可恢复/不可恢复的错误消息和资源利用。基于网络的工具正在开发捷豹XT5在橡树岭领导计算设施。
Analyzing system logs provides useful insights for identifying system/application anomalies and helps in better usage of system resources. Nevertheless, it is simply not practical to scan through the raw log messages on a regular basis for large-scale systems. First, the sheer volume of unstructured log messages affects the readability, and secondly correlating the log messages to system events is a daunting task. These factors limit large-scale system logs primarily for generating alerts on known system events, and post-mortem diagnosis for identifying previously unknown system events that impacted the systems performance. In this paper, we describe a log monitoring framework that enables prompt analysis of system events in real-time. Our web-based framework provides a summarized view of console, netwatch, consumer, and apsched logs in real- time. The logs are parsed and processed to generate views of applications, message types, individual/group of compute nodes, and in sections of the compute platform. Also from past application runs we build a statistical profile of user/application characteristics with respect to known system events, recoverable/non-recoverable error messages and resources utilized. The web-based tool is being developed for Jaguar XT5 at the Oak Ridge Leadership Computing facility.