Persistent Spread Measurement for Big Network Data Based on Register Intersection
Persistent Spread Measurement for Big Network Data Based on Register Intersection
复制标题
DOI：
10.1145/3084452
复制
发表时间：
2017-04
期刊：
影响因子：
--
通讯作者：
You Zhou;Yian Zhou;Min Chen;Shigang Chen
中科院分区：
文献类型：
--
作者：
You Zhou;Yian Zhou;Min Chen;Shigang Chen
持续传播度量是计算在预定义时间段内持续存在于每个网络流中的不同元素的数量。它有许多实际应用，包括在正常用户活动的背景下检测长期的隐蔽网络活动，如隐蔽的DDoS攻击、隐蔽的网络扫描或伪造的网络趋势，这些都是传统流量基数测量无法检测到的。对于大型网络数据，一个挑战是在不产生太多内存开销的情况下测量大量流的持续分布，因为这种测量可能由具有快速但小片上内存的网络处理器以线路速度执行。为此，我们提出了一个高度紧凑的虚拟交集HyperLogLog （VI-HLL）架构。该方法大大提高了v -位图的存储效率，同时大大扩展了测量范围。理论分析和大量实验表明，即使在每流小于1比特的非常紧张的存储空间中，VI-HLL也能提供良好的测量精度。
Persistent spread measurement is to count the number of distinct elements that persist in each network flow for predefined time periods. It has many practical applications, including detecting long-term stealthy network activities in the background of normal-user activities, such as stealthy DDoS attack, stealthy network scan, or faked network trend, which cannot be detected by traditional flow cardinality measurement. With big network data, one challenge is to measure the persistent spreads of a massive number of flows without incurring too much memory overhead as such measurement may be performed at the line speed by network processors with fast but small on-chip memory. We propose a highly compact Virtual Intersection HyperLogLog (VI-HLL) architecture for this purpose. It achieves far better memory efficiency than the best prior work of V-Bitmap, and in the meantime drastically extends the measurement range. Theoretical analysis and extensive experiments demonstrate that VI-HLL provides good measurement accuracy even in very tight memory space of less than 1 bit per flow.