Collaborative Research: CT-ISG: Accurate Sampling of the Internet for Effective Anomaly Detection

摘要：由于高速的链路速度使得不可能对每一个数据包进行检测，因此越来越多地使用采样流量数据作为异常检测系统的输入。这就提出了一个重要的问题，即采样是否对异常检测的准确性/有效性有（负面）影响，如果有，如何减轻这种影响。智力价值：本项目从以下三个角度系统研究上述问题。首先，我们将识别对各种异常检测方案至关重要的流量特征，并量化它们被各种采样方案扭曲的程度。其次，我们将设计新的采样或测量技术，以保持足够的精度来支持有效的异常检测，同时具有成本效益和重量轻。第三，我们将研究如何将在边缘路由器上获得的NetFlowsamples与使用现有数据流算法生成的信息丰富的数据相关联，以获得比纯采样更好的异常检测。通过这项研究获得的新科学知识将为我们提供更好的技术来监测大型高速网络的异常行为。更广泛的影响：结果将通过出版物、特邀演讲和教程以及为该项目开发的开源软件广泛传播。pi与一级ISP的合作将促进技术从研究环境转移到生产网络的实际管理。研究成果将纳入信息安全课程。这两个国家一直在积极吸引代表性不足的群体参与研究和高等教育，并将继续扩大这些努力。

Title: Collaborative Research: CT-ISG: Accurate Sampling of the Internetfor Effective Anomaly DetectionAbstract:Sampled traffic data has been increasingly used as input for anomalydetection systems, as the high link speeds make it impossible toexamine each and every packet. This raises an important question ofwhether sampling has a (negative) impact on the accuracy/effectivenessof anomaly detection, and if so how to mitigate this effect.Intellectual Merit: This project systematically studies the questionmentioned above from the following three angles. First, we willidentify traffic features that are critical for a wide range ofanomaly detection schemes and quantify how much they are distorted byvarious sampling schemes. Second, we will design new sampling ormeasurement techniques that preserve enough accuracy to supporteffective anomaly detection, while being cost-effective andlight-weight. Third, we will study how to correlate the NetFlowsamples obtained at the edge routers with the information-rich datagenerated using existing data streaming algorithms, for much betteranomaly detection than pure sampling. The new scientific knowledgelearned through this research will provide us with much bettertechnologies to monitor large high-speed networks for anomalousbehaviors.Broader impact: The results will be broadly disseminated throughpublications, invited talks and tutorials, and open-sourcing ofsoftware developed for this project. The PIs' collaboration withtier-1 ISP's will facilitate the transfer of technology from researchenvironment to actual managing of production networks. Researchresults will be incorporated into information security curriculum.Both PIs have been actively engaging under-represented groups inresearch and higher education and will continue and expand theseefforts.