WhatsThat? On the Usage of Hierarchical Clustering for Unsupervised Detection & Interpretation of Network Attacks
WhatsThat? On the Usage of Hierarchical Clustering for Unsupervised Detection & Interpretation of Network Attacks
复制标题
DOI：
10.1109/eurospw51379.2020.00084
复制
发表时间：
2020-09
期刊：
影响因子：
--
通讯作者：
Pavol Mulinka;K. Fukuda;P. Casas;L. Kencl
中科院分区：
文献类型：
--
作者：
Pavol Mulinka;K. Fukuda;P. Casas;L. Kencl
通过机器学习自动检测和解释网络攻击是一个众所周知的问题，目前还没有通用的解决方案。监督学习和异常检测方法需要对被分析系统的先验知识，或者以正常操作概要的形式，或者以要检测的特定攻击的形式。因此，这两种方法在检测，特别是解释以前未见过的攻击和异常时都有明显的局限性。在本文中，我们提出了一种新的无监督网络异常检测方法WhatsThat，它可以以完全黑盒的方式检测和解释异常行为，而不依赖于分析系统的任何基本事实。它依赖于分层聚类技术来发现和描述嵌套或分层结构多维数据中存在的异常模式，这在网络流量中很常见-例如，由于多层协议。该解决方案使用无监督簇有效性度量来自动探索数据结构，并建立在自动识别相关特征的基础上，为检测到的模式提供有意义的描述。我们展示了WhatsThat在检测和解释隐藏在真实的网络攻击中，在传输互联网骨干网收集大规模网络流量。虽然WhatsThat主要是为无监督异常检测和解释量身定制的，但它也可以应用于任何类型的嵌套或分层结构多维数据的无监督分析，显示了分层聚类在一般无监督数据分析中的潜力。
The automatic detection and interpretation of network attacks through machine learning is a well-known problem, for which no general solution is available. Super-vised learning and anomaly detection approaches require prior knowledge on the system under analysis, either in the form of normal operation profiles, or on the specific attacks to detect. As a consequence, both approaches have clear limitations when it comes to detecting, and in particular interpreting, previously unseen attacks and anomalies.In this paper we present WhatsThat, a novel approach to unsupervised network anomaly detection, which can both detect and interpret anomalous behaviors in a completely black-box manner, without relying on any ground-truth on the system under analysis. WhatsThat relies on hierarchical–clustering techniques to discover and characterize anomalous patterns present in nested or hierarchically structured multi–dimensional data, which is common in network traffic – e.g., due to multi–layer protocols. The solution uses unsupervised cluster validity metrics to automatically explore the data structure, and builds on automatic identification of relevant features to provide meaningful descriptions for the detected patterns. We showcase WhatsThat in the detection and interpretation of network attacks hidden in real, large-scale network traffic collected at a transit Internet backbone network. While WhatsThat is mainly tailored for unsupervised anomaly detection and interpretation, it can also be applied to the unsupervised analysis of any kind of nested or hierarchically structured multi-dimensional data, showing the potential of hierarchical clustering for general unsupervised data analysis.