HUMAN - Hierarchical Clustering for Unsupervised Anomaly Detection and Interpretation
HUMAN - Hierarchical Clustering for Unsupervised Anomaly Detection and Interpretation
复制标题
HUMAN - 用于无监督异常检测和解释的层次聚类
DOI：
10.1109/nof50125.2020.9249194
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
L.Kencl
中科院分区：
文献类型：
--
作者：
P.Mulinka;P.Casas;K.Fukuda;L.Kencl
通过机器学习自动检测和解释网络流量异常是一个众所周知的问题，目前还没有通用的解决方案。监督和非监督（即异常检测）方法都需要事先了解监测数据，无论是在正常操作概况方面还是在要检测的特定异常方面。因此，这两种方法在探测，特别是解释以前看不见的事件时都有明显的局限性。我们提出了HUMAN，一种通用的基于分层聚类的无监督网络流量分析方法，它可以以完全黑盒的方式检测和解释异常行为，而不依赖于分析系统的真实情况。HUMAN可以使用利用层次聚类关系和特征之间的相关性的结构方法来检测和解释分析数据中的复杂模式。我们描述了HUMAN的构建模块，并详细解释了它的功能，通过对分布式主动云延迟测量的无监督分析，使用主要云平台中性能问题的检测和解释作为案例研究。HUMAN方法可以应用于任何类型的嵌套或分层结构多维数据的无监督分析，显示了一般无监督数据分析的分层聚类潜力。
The automatic detection and interpretation of network traffic anomalies through machine learning is a well-known problem, for which no general solution is available. Both supervised and unsupervised (i.e., anomaly detection) approaches require prior knowledge on the monitoring data, either in terms of normal operation profiles or on the specific anomalies to detect. As a consequence, both approaches have clear limitations when it comes to detecting, and in particular interpreting, previously unseen events. We present HUMAN, a general hierarchical-clustering-based approach for unsupervised network traffic analysis, which can both detect and interpret anomalous behaviors in a completely black-box manner, without relying on ground-truth on the system under analysis. HUMAN can detect and interpret complex patterns in the analyzed data, using a structural approach which exploits hierarchical cluster relationships and correlation among features. We describe the building blocks of HUMAN and explain its functioning in detail, using as case study the detection and interpretation of performance issues in major cloud platforms, through the unsupervised analysis of distributed active cloud latency measurements. The HUMAN approach can be applied to the unsupervised analysis of any kind of nested or hierarchically structured multi-dimensional data, showing the potential of hierarchical clustering for general unsupervised data analytics.