Collaborative Research: CISE-ANR: CNS Core: Small: Modeling Modern Network Traffic: From Data Representation to Automated Machine Learning

为了成功维护和保护通信网络，运营商需要监控其行为，并在出现安全性、性能和其他问题时进行调查。网络协议和应用的最新进展为监控网络流量提出了根本性的挑战。具体而言，互联网流量，从Web流量到域名系统（DNS）查询和响应，正在变得无处不在地加密，混淆了原本可能用于这些任务的信息。 此外，网络流量的数量和速率都在增加，无法详细记录和分析单个数据包或数据流。最后，互联网变得更加集中，许多服务也变得基于云，这使得根据IP地址和端口号等固定标识符识别应用程序或服务变得更加困难。 因此，回答有关互联网流量的基本问题也变得越来越具有挑战性。该项目旨在开发技术，以重新获得对考虑这些趋势的现代网络流量的可见性和洞察力。 我们解决三个研究问题，以恢复到现代网络流量的可见性。首先，该项目将研究如何以适合建模的方式表示交通数据，并且可以优化有监督和无监督建模任务的模型。我们将探讨表示在四个维度上的影响：（1）时间序列表示;（2）跨流表示;（3）更高层表示;（4）压缩数据上的操作。 其次，我们将在流量数据表示的基础上开发一套工具，用于自动探索为网络流量问题量身定制的模型和流量表示。为了实现这一目标，我们将在几个不同的应用程序和服务中构建一个大规模的标记流存储库，并评估用于构建网络流量统计学习模型的数据表示。最后，我们将使用我们构建的软件平台和算法为运营商设计新的技术和工具，以解决阻止他们将开发的模型从实验室实验转移到现实世界部署的挑战。我们将扩展自动模型选择，以考虑系统成本和现实世界的限制;解决需要能够确定模型何时变得不准确，并将模型不准确与网络固有的问题区分开来;并通过研究模型转移的一般方法来提高模型的鲁棒性。 我们在这个项目中创建的所有软件都将是公开的和开源的。此外，我们计划将软件系统集成到社区教程、本科生和研究生课程以及社区的推广和教育计划中，与芝加哥大学特别项目办公室和公民参与办公室等合作伙伴合作。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的评估来支持。影响审查标准。

To successfully maintain and secure communications networks, operators need to monitor their behavior and investigate security, performance, and other problems as they arise. Recent advances in network protocols and applications present fundamental challenges for monitoring network traffic. Specifically, Internet traffic, from web traffic to Domain Name System (DNS) queries and responses, is becoming ubiquitously encrypted, obfuscating information that might otherwise be available for these tasks. Additionally, network traffic is increasing in volume and rate, precluding detailed logging and analyzing individual packets or streams. Finally, the Internet is becoming more centralized, and many services have also become cloud-based, making it more difficult to identify applications or services according to fixed identifiers such as IP addresses and port numbers. Answering even basic questions about Internet traffic has thus become increasingly challenging. This project seeks to develop techniques to regain visibility and insights into modern network traffic considering these trends. We address three research questions towards regaining visibility into modern network traffic. First, this project will study how to represent traffic data in ways that are amenable to modeling, and that could optimize models for both supervised and unsupervised modeling tasks. We will explore the impact of representations across four dimensions: (1) timeseries representations; (2) representations across flows; (3) representations at higher layers; and (4) operations on compressed data. Second, we will build on our work on traffic data representation to develop a set of tools to automatically explore model and traffic representations tailored for network traffic problems. Towards this goal, we will build a large-scale repository of labeled flows across several different applications and services as well as evaluate data representations that will be used to build statistical learning models about network traffic. Finally, we will use the software platforms and algorithms we build to design new techniques and tools for operators to solve the challenges that prevent them from transferring developed models from laboratory experiments to real-world deployments. We will extend automated model selection to account for systems costs and real-world limitations; address the need to be able to determine when models become inaccurate and to distinguish model inaccuracies from problems that are inherent to the network; and improve model robustness by investigating general approaches for model transfer. All software we create in this project will be publicly available and open source. Additionally, we plan to integrate the software systems into tutorials for the community, undergraduate and graduate courses, and outreach and education programs in the community, in collaboration with partners such as the University of Chicago's Office of Special Programs and Office of Civic Engagement.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.