Drill: Log-based Anomaly Detection for Large-scale Storage Systems Using Source Code Analysis
Drill: Log-based Anomaly Detection for Large-scale Storage Systems Using Source Code Analysis
复制标题
DOI：
10.1109/ipdps54959.2023.00028
复制
发表时间：
2023-05
期刊：
影响因子：
--
通讯作者：
Di Zhang;Chris Egersdoerfer;Tabassum Mahmud;Mai Zheng;Dong Dai
中科院分区：
文献类型：
--
作者：
Di Zhang;Chris Egersdoerfer;Tabassum Mahmud;Mai Zheng;Dong Dai
大规模存储系统是现代计算系统的关键部分，受生产中的各种运行时错误，故障和异常。因此，在运行时识别其异常对用户和管理员至关重要。由于运行时日志记录了系统的重要状态，因此已经对基于日志的异常检测进行了广泛的研究，以及时识别系统故障。但是，现有的基于日志的异常检测解决方案在准确，稳健地表示日志条目方面具有共同的限制，因此无法有效处理历史日志中未见的日志条目，这是由于日志固有的稀有性而引起的常见现实世界情景以及系统的持续发展。为了解决现有方法的问题，我们提出了钻探，这是一种新的日志预处理方法，可以通过利用两个存储系统特定的情感分类语言模型和从源代码构建的存储系统特定的情感分类语言模型和日志上下文来生成运行时日志的高质量矢量表示。通过对两个代表性分布式存储系统（Apache HDFS和Luster）的广泛评估，我们表明，与最先进的异常检测解决方案相比，钻头可以提高41％检测。
Large-scale storage systems, a critical part of modern computing systems, are subject to various runtime bugs, failures, and anomalies in production. Identifying their anomalies at runtime is thus critical for users and administrators. Since runtime logs record the important status of the systems, log-based anomaly detection has been studied extensively for timely identifying system malfunctions. However, existing log-based anomaly detection solutions share common limitations in representing log entries accurately and robustly, hence can not effectively handle log entries that were not seen in the historical logs, which is a common real-world scenario due to logs' inherent rarity and the continuous evolution of the systems. To address the issues of existing methods, we propose Drill, a new log pre-processing method to generate high-quality vector representation of runtime logs by leveraging both storage system-specific sentiment-classifying language models and log contexts built from the source code. Through extensive evaluations of two representative distributed storage systems (Apache HDFS and Lustre), we show that Drill can achieve up to 41% improvement when compared with state-of-the-art anomaly detection solutions, showing it is a promising solution for general anomaly detection.