NeTS-NBD: Accurate Estimation of Network Measurement Matrices Using Multiple Data Sources

流量和损失矩阵是最重要的网络性能统计数据，需要在互联网服务提供商（ISP）网络中进行准确测量和仔细监控。 它们对于网络管理功能（如流量工程、容量规划、流量异常检测和故障诊断）至关重要。 用于估计流量矩阵的现有技术是从可以直接测量的一些其他网络统计数据（诸如SNMP链路计数或Cisco NetFlow记录）间接地推断流量矩阵。 然而，这些推断技术的准确性通常不是很高，因为这样的统计数据可能是嘈杂的、稀疏的和肮脏的。 我们建议开发新的统计信号处理技术，使我们能够尽可能准确地从我们已经掌握的数据中推断出流量和损失矩阵，这些数据可能是嘈杂的，稀疏的和肮脏的。一般方法将多个数据源关联起来，通过利用独立观测中噪声的统计正交性质，获得比从单个数据源获得的更好的准确性，并识别和删除脏数据。 更广泛的影响：该项目将为本科生和研究生提供跨多个学科的研究和学习经验。 通过该项目与工业界的持续合作将促进科学发现在应用领域的应用。研究结果将通过论文、讲座、研讨会和软件发布等方式广泛传播。 研究所将继续努力，积极吸引代表性不足的群体参与研究和教育。

Traffic and loss matrices are the most important network performance statistics that need to be accurately measured and carefully monitored in an Internet service provider (ISP) network. They are essential for network management functions such as traffic engineering, capacity planning, traffic anomaly detection, and fault diagnosis. Existing techniques for estimating traffic matrix are to infer it indirectly from some other network statistics that can be directly measured such as SNMP link counts or Cisco NetFlow records. The accuracy of these inference techniques, however, are generally not very high because such statistics can be noisy, sparse, and dirty. We propose to develop novel statistical signal processing techniques that allow us to infer traffic and loss matrices as accurately as possible from the data we already have in hand that can be noisy, sparse, and dirty.The general methodology correlates multiple sources of data, which by exploiting the statistically orthogonal nature of noises in independent observations, achieves much better accuracy than obtainable from a single data source and identifies and removes dirty data. Broader Impacts: This project will offer undergraduate and graduate students research and learning experience across multiple disciplines. The ongoing collaborations with industry through this project will facilitate application of scientific discoveries to the application domains. The results will be broadly disseminated through papers, talks, workshops, and software releases. The PIs will continue to work hard to actively engage underrepresented groups in research and education.