IMR: MM-1C: Fine-grained Network Monitoring via Software Imputation

计算机网络是现代社会中驱动众多产品和服务的计算基础设施的重要组成部分。网络监控对于检测恶意活动、故障排除和管理网络资源至关重要。然而，由于硬件的限制，精确的监测是非常昂贵的，甚至是不可行的。网络运营商使用采样，即，它们较不频繁地监视网络以节省资源。然而，采样使网络管理更具挑战性，因为它可能隐藏重要的见解或错过某些事件。该项目将开发创新技术，以构建一个软件组件，即遥测估算层（TIL），它将在网络硬件上工作，以提高监测的准确性。TIL有可能彻底改变网络管理，网络运营商将获得前所未有的质量监控，从而促进更安全，可靠和高性能的网络。在高层次上，TIL类似于图像超分辨率，其中低分辨率图像可以变成高分辨率图像。对于图像，由于相邻像素之间的相关性和图像的底层结构，超分辨率是可能的。对于网络监控，由于存在物理约束和监控时间序列之间的相关性，因此可以进行插补，这项研究涉及解决跨学科的挑战，需要系统，网络，机器学习（ML）和形式化方法（FM）的知识，以促进网络监控的进步。首先，这项研究将开发一个ML模型，从粗粒度的测量中恢复细粒度的监控数据，精确到足以执行已知的网络管理任务。为此，该研究将研究不同的机器学习模型和训练管道，以避免常见的机器学习陷阱，例如缺乏通用性，过度拟合和数据稀缺。接下来，本研究将开发FM技术和基于逻辑的模型，通过约束连接网络操作和监测测量。使用此模型，该项目将提供使用与给定场景和粗粒度测量一致的细粒度网络数据来回答网络管理查询的方法。最后，该项目旨在开发将ML和FM技术联合收割机用于网络估算的方法，以便从网络领域的数据和知识中受益。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Computer networks are an essential component of the computing infrastructure that drives numerous products and services in modern society. Network monitoring is essential for detecting malicious activities, troubleshooting, and managing the resources of a network. However, accurate monitoring is notoriously expensive or even infeasible, due to hardware limitations. Network operators use sampling, i.e., they monitor the network less frequently to save on resources. However, sampling makes network management more challenging as it can hide important insights or miss certain events. This project will develop innovative technologies to build a software component, namely a Telemetry Imputation Layer (TIL), that will work atop the networking hardware to improve the accuracy of monitoring. TIL has the potential to revolutionize network management, where network operators will have access to monitoring of unprecedented quality, thereby facilitating more secure, reliable, and performant networks. At a high level, TIL is analogous to image super-resolution in which low-resolution images can be turned into high-resolution ones. For images, super-resolution is possible thanks to the correlations among neighboring pixels and the underlying structure of the images. For network monitoring, the imputation is possible due to the existence of physical constraints and of correlations among the monitored time series.This research involves solving interdisciplinary challenges that require knowledge of systems, networking, machine learning (ML), and formal methods (FM), to facilitate advances in network monitoring. First, this research will develop an ML model that recovers fine-grained monitoringdata from coarse-grained measurements, precisely enough to perform known network management tasks. To this end, the research will investigate different ML models and training pipelines to avoid common ML pitfalls such as lack of generality, overfitting, and data scarcity. Next, this research will develop FM techniques and a logic-based model that connects network operations and monitored measurements via constraints. Using this model, the project will provide the means to answer network management queries using fine-grained network data that are consistent with given scenarios and coarse-grained measurements. Finally, this project aims to develop methods that combine the ML and FM techniques for network imputation in order to benefit from both the existence of data and knowledge in the networking domain.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.