An Efficient Network Log Anomaly Detection System Using Random Projection Dimensionality Reduction
An Efficient Network Log Anomaly Detection System Using Random Projection Dimensionality Reduction
复制标题
DOI：
10.1109/ntms.2014.6814006
复制
发表时间：
2014-05
期刊：
影响因子：
--
通讯作者：
Antti Juvonen;T. Hämäläinen
中科院分区：
文献类型：
--
作者：
Antti Juvonen;T. Hämäläinen
网络流量一直在增加，网络服务也变得越来越复杂和脆弱。为了保护这些网络，需要使用入侵检测系统。基于签名的入侵检测无法发现以前未知的攻击，因此需要进行异常检测。然而，许多新系统是缓慢和复杂的。我们提出了一个日志异常检测框架，旨在促进快速异常检测，并提供网络流量结构的可视化。该系统将网络日志预处理成一个数值数据矩阵，使用随机投影降低该矩阵的维数，并使用马氏距离找到异常值并计算每个数据点的异常分数。差异太大的日志行被标记为异常。通过对真实网络数据的测试，发现了实际的入侵企图。此外，还创建了可视化来表示网络数据的结构。我们还进行了计算时间评估，以确保性能是可行的。该系统速度快，发现入侵企图，并且不需要干净的训练数据。
Network traffic is increasing all the time and network services are becoming more complex and vulnerable. To protect these networks, intrusion detection systems are used. Signature-based intrusion detection cannot find previously unknown attacks, which is why anomaly detection is needed. However, many new systems are slow and complicated. We propose a log anomaly detection framework which aims to facilitate quick anomaly detection and also provide visualizations of the network traffic structure. The system preprocesses network logs into a numerical data matrix, reduces the dimensionality of this matrix using random projection and uses Mahalanobis distance to find outliers and calculate an anomaly score for each data point. Log lines that are too different are flagged as anomalies. The system is tested with real-world network data, and actual intrusion attempts are found. In addition, visualizations are created to represent the structure of the network data. We also perform computational time evaluation to ensure the performance is feasible. The system is fast, finds intrusion attempts and does not need clean training data.