Statistical Methods for Big Network Flow Data Analysis

估计网络流量矩阵在当今的流量管理、网络规模确定和异常检测中起着至关重要的作用。存在几种方法用于恢复网络流量矩阵，但是对于运营商来说，有效地估计网络流量矩阵以达到现代流量管理系统所需的精确粒度水平仍然是一项具有挑战性的任务。主要有三个方面需要改进。首先，虽然有技术可以在短时间内检测到网络异常，例如拒绝服务攻击，但运营商仍然需要很长时间才能恢复正常运行的网络状态。这会导致意外的服务延迟，并且难以提供服务质量保证。因此，需要更先进的网络流量表示和预测方法来支持在流量事件或恶意攻击后的短时间内恢复网络状态。其次，我们需要找到可扩展的方法，对数据中的噪声具有鲁棒性，并改进降维方法以克服“高维灾难”。思科预测，到2021年，全球每月移动的数据流量将达到49 EB。这肯定会通过增加噪声和更多的因变量使网络流量数据的分析更具挑战性。第三，当前的技术水平主要依赖于网络业务流的传统表示，诸如通过在某些网络位置处的聚合业务计数，这提供了不足以支持现代业务管理系统的要求的粒度。因此，需要更多的工作来开发可以从大网络流数据中提取可操作见解的方法，这些方法在多样性，准确性和数量方面具有复杂性。

Estimating network traffic matrices plays a crucial role in today's traffic management, network dimensioning, and anomaly detection. Several methods exist for recovery of network traffic matrices, but it is still a challenging task for operators to efficiently estimate network traffic matrices to the level of accuracy granularity required by modern traffic management systems. There are three main aspects that need to be improved. First, although there are technologies that can detect network anomalies within a short time, such as denial of service attack, it still takes long time for operators to recover a well-functioning network state. This causes unexpected service delays and makes difficult to provide quality of services guarantees. Therefore, more advanced methods for network traffic representation and prediction are required to support the recovery of network state within a short time period after a traffic incident or a malicious attack. Second, we need to find methods that are scalable and robust to noise in the data and improve upon the dimensionality reduction methods to overcome the 'curse of high-dimensionality.' Cisco forecasts that the monthly global mobile data traffic will be 49 Exabytes by 2021. This will certainly make the analysis of network flow data more challenging through added noise and more dependent variables. Third, the current state-of-the-art largely rests on traditional representations of network traffic flows, such as by aggregate traffic counts at some network locations, which provides insufficient granularity to support requirements of modern traffic management systems. Therefore, more work is required to develop methods that can extract actionable insights from big network flow data, having complexity in variety, veracity and volume.