Collaborative Research: CISE-ANR: CNS Core: Small: Modeling Modern Network Traffic: From Data Representation to Automated Machine Learning

为了成功维护和保护通信网络，运营商需要监控其行为，并在出现安全、性能和其他问题时进行调查。网络协议和应用的最新进展对监控网络流量提出了根本挑战。具体地说，从Web流量到域名系统(DNS)查询和响应的互联网流量正变得无处不在地加密，混淆了原本可能可用于这些任务的信息。此外，网络流量在数量和速率上都在增加，因此无法详细记录和分析单个数据包或流。最后，互联网正变得更加集中化，许多服务也变得基于云，这使得根据IP地址和端口号等固定标识识别应用或服务变得更加困难。因此，即使是回答有关互联网流量的基本问题也变得越来越具有挑战性。考虑到这些趋势，该项目寻求开发技术以重新获得对现代网络流量的可见性和洞察力。为了重新获得对现代网络流量的可见性，我们解决了三个研究问题。首先，这个项目将研究如何以适合建模的方式来表示交通数据，并且可以为监督和非监督建模任务优化模型。我们将探讨表示法在四个维度上的影响：(1)时间序列表示法；(2)跨流表示法；(3)更高层的表示法；以及(4)对压缩数据的操作。其次，我们将在流量数据表示的基础上开发一套工具，以自动探索针对网络流量问题量身定做的模型和流量表示。为了实现这一目标，我们将构建跨几个不同应用程序和服务的标记流的大规模存储库，并评估将用于构建有关网络流量的统计学习模型的数据表示法。最后，我们将使用我们构建的软件平台和算法为运营商设计新的技术和工具，以解决阻止他们将开发的模型从实验室实验转移到现实世界部署的挑战。我们将扩展自动模型选择，以考虑系统成本和现实世界的限制；解决能够确定模型何时变得不准确的需求，并将模型不准确与网络固有的问题区分开来；通过研究模型传输的一般方法来提高模型的稳健性。我们在这个项目中创建的所有软件都将是公开可用的和开源的。此外，我们计划与芝加哥大学特别项目办公室和公民参与办公室等合作伙伴合作，将软件系统整合到社区教程、本科生和研究生课程以及社区外展和教育项目中。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

To successfully maintain and secure communications networks, operators need to monitor their behavior and investigate security, performance, and other problems as they arise. Recent advances in network protocols and applications present fundamental challenges for monitoring network traffic. Specifically, Internet traffic, from web traffic to Domain Name System (DNS) queries and responses, is becoming ubiquitously encrypted, obfuscating information that might otherwise be available for these tasks. Additionally, network traffic is increasing in volume and rate, precluding detailed logging and analyzing individual packets or streams. Finally, the Internet is becoming more centralized, and many services have also become cloud-based, making it more difficult to identify applications or services according to fixed identifiers such as IP addresses and port numbers. Answering even basic questions about Internet traffic has thus become increasingly challenging. This project seeks to develop techniques to regain visibility and insights into modern network traffic considering these trends. We address three research questions towards regaining visibility into modern network traffic. First, this project will study how to represent traffic data in ways that are amenable to modeling, and that could optimize models for both supervised and unsupervised modeling tasks. We will explore the impact of representations across four dimensions: (1) timeseries representations; (2) representations across flows; (3) representations at higher layers; and (4) operations on compressed data. Second, we will build on our work on traffic data representation to develop a set of tools to automatically explore model and traffic representations tailored for network traffic problems. Towards this goal, we will build a large-scale repository of labeled flows across several different applications and services as well as evaluate data representations that will be used to build statistical learning models about network traffic. Finally, we will use the software platforms and algorithms we build to design new techniques and tools for operators to solve the challenges that prevent them from transferring developed models from laboratory experiments to real-world deployments. We will extend automated model selection to account for systems costs and real-world limitations; address the need to be able to determine when models become inaccurate and to distinguish model inaccuracies from problems that are inherent to the network; and improve model robustness by investigating general approaches for model transfer. All software we create in this project will be publicly available and open source. Additionally, we plan to integrate the software systems into tutorials for the community, undergraduate and graduate courses, and outreach and education programs in the community, in collaboration with partners such as the University of Chicago's Office of Special Programs and Office of Civic Engagement.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.