Collaborative Research: CT-ISG: Accurate Sampling of the Internet for Effective Anomaly Detection

职务名称：合作研究：CT-ISG：准确采样的互联网有效的异常检测摘要：采样的流量数据已越来越多地用作输入异常检测系统，因为高链路速度使它不可能检查每一个数据包。 这就提出了一个重要的问题，即采样是否对异常检测的准确性/有效性有（负面）影响，如果有，如何减轻这种影响。智力成果：本项目从以下三个角度系统地研究了上述问题。 首先，我们将确定的流量功能，是非常重要的范围ofanomaly检测方案和量化多少，他们被扭曲的各种抽样方案。 其次，我们将设计新的采样或测量技术，保持足够的准确性，以有效的异常检测，同时具有成本效益和重量轻。 第三，我们将研究如何将在边缘路由器处获得的NetFlow样本与使用现有数据流算法生成的信息丰富的数据相关联，以获得比纯采样更好的异常检测。 通过这项研究获得的新的科学知识将为我们提供更好的技术来监控大型高速网络中的恶意行为。更广泛的影响：研究结果将通过出版物、特邀演讲和教程以及为该项目开发的开源软件广泛传播。 PI与一级ISP的合作将促进技术从研究环境转移到生产网络的实际管理。 研究成果将被纳入信息安全课程。两个PI一直积极参与研究和高等教育的代表性不足的群体，并将继续和扩大这些努力。

Title: Collaborative Research: CT-ISG: Accurate Sampling of the Internetfor Effective Anomaly DetectionAbstract:Sampled traffic data has been increasingly used as input for anomalydetection systems, as the high link speeds make it impossible toexamine each and every packet. This raises an important question ofwhether sampling has a (negative) impact on the accuracy/effectivenessof anomaly detection, and if so how to mitigate this effect.Intellectual Merit: This project systematically studies the questionmentioned above from the following three angles. First, we willidentify traffic features that are critical for a wide range ofanomaly detection schemes and quantify how much they are distorted byvarious sampling schemes. Second, we will design new sampling ormeasurement techniques that preserve enough accuracy to supporteffective anomaly detection, while being cost-effective andlight-weight. Third, we will study how to correlate the NetFlowsamples obtained at the edge routers with the information-rich datagenerated using existing data streaming algorithms, for much betteranomaly detection than pure sampling. The new scientific knowledgelearned through this research will provide us with much bettertechnologies to monitor large high-speed networks for anomalousbehaviors.Broader impact: The results will be broadly disseminated throughpublications, invited talks and tutorials, and open-sourcing ofsoftware developed for this project. The PIs' collaboration withtier-1 ISP's will facilitate the transfer of technology from researchenvironment to actual managing of production networks. Researchresults will be incorporated into information security curriculum.Both PIs have been actively engaging under-represented groups inresearch and higher education and will continue and expand theseefforts.