Statistical Methods for Big Network Flow Data Analysis

点击翻译按钮获取中文摘要

Estimating network traffic matrices plays a crucial role in today's traffic management, network dimensioning, and anomaly detection. Several methods exist for recovery of network traffic matrices, but it is still a challenging task for operators to efficiently estimate network traffic matrices to the level of accuracy granularity required by modern traffic management systems. There are three main aspects that need to be improved. First, although there are technologies that can detect network anomalies within a short time, such as denial of service attack, it still takes long time for operators to recover a well-functioning network state. This causes unexpected service delays and makes difficult to provide quality of services guarantees. Therefore, more advanced methods for network traffic representation and prediction are required to support the recovery of network state within a short time period after a traffic incident or a malicious attack. Second, we need to find methods that are scalable and robust to noise in the data and improve upon the dimensionality reduction methods to overcome the 'curse of high-dimensionality.' Cisco forecasts that the monthly global mobile data traffic will be 49 Exabytes by 2021. This will certainly make the analysis of network flow data more challenging through added noise and more dependent variables. Third, the current state-of-the-art largely rests on traditional representations of network traffic flows, such as by aggregate traffic counts at some network locations, which provides insufficient granularity to support requirements of modern traffic management systems. Therefore, more work is required to develop methods that can extract actionable insights from big network flow data, having complexity in variety, veracity and volume.