NeTS-NBD: Accurate Estimation of Network Measurement Matrices Using Multiple Data Sources

流量和损失矩阵是最重要的网络性能统计数据，需要在Internet服务提供商（ISP）网络中进行精确测量和仔细监控。它们是流量工程、容量规划、流量异常检测和故障诊断等网络管理功能的基础。现有的估算流量矩阵的技术是从其他一些可以直接测量的网络统计数据（如SNMP链路计数或Cisco NetFlow记录）中间接推断出来的。然而，这些推断技术的准确性通常不是很高，因为这些统计数据可能是嘈杂的、稀疏的和肮脏的。我们建议开发新的统计信号处理技术，使我们能够尽可能准确地从我们已经掌握的数据中推断交通和损失矩阵，这些数据可能是嘈杂的、稀疏的和脏的。一般方法将多个数据源关联起来，通过利用独立观测中噪声的统计正交特性，获得比单一数据源更好的准确性，并识别和去除脏数据。更广泛的影响：该项目将为本科生和研究生提供跨学科的研究和学习经验。通过该项目与工业界的持续合作将促进科学发现在应用领域的应用。研究结果将通过论文、演讲、研讨会和软件发布广泛传播。pi将继续努力让代表性不足的群体积极参与研究和教育。

Traffic and loss matrices are the most important network performance statistics that need to be accurately measured and carefully monitored in an Internet service provider (ISP) network. They are essential for network management functions such as traffic engineering, capacity planning, traffic anomaly detection, and fault diagnosis. Existing techniques for estimating traffic matrix are to infer it indirectly from some other network statistics that can be directly measured such as SNMP link counts or Cisco NetFlow records. The accuracy of these inference techniques, however, are generally not very high because such statistics can be noisy, sparse, and dirty. We propose to develop novel statistical signal processing techniques that allow us to infer traffic and loss matrices as accurately as possible from the data we already have in hand that can be noisy, sparse, and dirty.The general methodology correlates multiple sources of data, which by exploiting the statistically orthogonal nature of noises in independent observations, achieves much better accuracy than obtainable from a single data source and identifies and removes dirty data. Broader Impacts: This project will offer undergraduate and graduate students research and learning experience across multiple disciplines. The ongoing collaborations with industry through this project will facilitate application of scientific discoveries to the application domains. The results will be broadly disseminated through papers, talks, workshops, and software releases. The PIs will continue to work hard to actively engage underrepresented groups in research and education.