Latent Variable Based Anomaly Detection in Network System Logs
Latent Variable Based Anomaly Detection in Network System Logs
复制标题
DOI：
10.1587/transinf.2018ofp0007
复制
发表时间：
2019-09
期刊：
影响因子：
--
通讯作者：
Kazuki Otomo;Satoru Kobayashi;K. Fukuda;H. Esaki
中科院分区：
文献类型：
--
作者：
Kazuki Otomo;Satoru Kobayashi;K. Fukuda;H. Esaki
系统日志对于了解大规模网络的状态和检测故障非常有用。然而，由于这些日志的多样性和数量，日志分析需要大量的时间和精力。本文提出了一种不需要预处理和特征提取的大规模网络日志事件异常检测方法。其核心思想是通过使用潜变量将大量不同的数据嵌入隐藏状态。我们评估我们的方法与12个月的系统日志从日本全国性的学术网络。通过与Kleinberg的单变量爆发检测和传统的多变量分析（即，PCA），我们证明了我们提出的方法实现了14.5%的召回率和3%的精度比PCA。实例分析表明，检测到的异常是网络系统故障排除的有效信息。关键词：网络操作，系统日志，系统日志，异常检测，潜变量分析，变分自动编码器
System logs are useful to understand the status of and detect faults in large scale networks. However, due to their diversity and volume of these logs, log analysis requires much time and effort. In this paper, we propose a log event anomaly detection method for large-scale networks without pre-processing and feature extraction. The key idea is to embed a large amount of diverse data into hidden states by using latent variables. We evaluate our method with 12 months of system logs obtained from a nation-wide academic network in Japan. Through comparisons with Kleinberg’s univariate burst detection and a traditional multivariate analysis (i.e., PCA), we demonstrate that our proposed method achieves 14.5% higher recall and 3% higher precision than PCA. A case study shows detected anomalies are effective information for troubleshooting of network system faults. key words: network operation, system logs, syslog, anomaly detection, latent variable analysis, variational autoencoder