Hashdoop: A MapReduce framework for network anomaly detection
Hashdoop: A MapReduce framework for network anomaly detection
复制标题
DOI：
10.1109/infcomw.2014.6849281
复制
发表时间：
2014-07
期刊：
影响因子：
--
通讯作者：
Romain Fontugne;J. Mazel;K. Fukuda
中科院分区：
文献类型：
--
作者：
Romain Fontugne;J. Mazel;K. Fukuda
异常检测对于防止网络中断和维护可用的网络资源至关重要。然而，为了应对互联网流量的不断增长，网络异常检测器仅暴露于采样流量，因此有害流量可能会逃避检测器检查。在本文中，我们研究了最新的分布式计算方法在实时分析非采样互联网流量方面的优势。我们的研究重点关注 MapReduce 模型，揭示了使用 Hadoop 检测网络流量异常的根本困难。由于 MapReduce 要求将数据集分为小块，并且异常检测器根据空间和时间流量结构计算统计数据，因此在分割流量时应特别小心。我们提出了 Hashdoop，这是一个 MapReduce 框架，它使用哈希函数分割流量以保留流量结构，从而保留分布式计算基础设施的利润来检测网络异常。 Hashdoop 的优势通过两个异常检测器和 2001 年至 2013 年间捕获的 15 个互联网骨干流量痕迹进行了评估。使用 6 节点集群 Hashdoop 提高了最慢检测器的吞吐量，速度提高了 15；因此，可以实时检测最大的分析痕迹。 Hashdoop 还提高了探测器的整体准确性，因为分裂通过减少周围的流量来强调异常情况。
Anomaly detection is essential for preventing network outages and maintaining the network resources available. However, to cope with the increasing growth of Internet traffic, network anomaly detectors are only exposed to sampled traffic, so harmful traffic may avoid detector examination. In this paper, we investigate the benefits of recent distributed computing approaches for real-time analysis of non-sampled Internet traffic. Focusing on the MapReduce model, our study uncovers a fundamental difficulty in order to detect network traffic anomalies by using Hadoop. Since MapReduce requires the dataset to be divided into small splits and anomaly detectors compute statistics from spatial and temporal traffic structures, special care should be taken when splitting traffic. We propose Hashdoop, a MapReduce framework that splits traffic with a hash function to preserve traffic structures and, hence, profits of distributed computing infrastructures to detect network anomalies. The benefits of Hashdoop are evaluated with two anomaly detectors and fifteen traces of Internet backbone traffic captured between 2001 and 2013. Using a 6-node cluster Hashdoop increased the throughput of the slowest detector with a speed-up of 15; thus, enabling real-time detection for the largest analyzed traces. Hashdoop also improved the overall detectors accuracy as splits emphasized anomalies by reducing the surrounding traffic.