NeTS-NBD: Network Data Streaming for Measurement and Monitoring of Future High-Speed Networks

准确的流量测量和监控是网络管理、运营和控制的关键。随着互联网的快速发展，网络连接速度每年都在加快，以容纳更多的互联网用户。对高速链路上的流量进行测量和监控已经成为一个非常具有挑战性的问题。首先在数据库领域引入的数据流被吹捧为解决这个问题的可行解决方案。数据流涉及使用一个小的工作存储器在一次传递中处理一长串数据项，以便回答关于该流的查询。挑战在于使用这个小内存来“记住”尽可能多的与查询相关的信息。然而，传统的数据流算法主要是为数据库应用而设计的，不适合未来的网络环境，因为未来的网络环境需要对大量数据流经众多高速链路进行监控。这是由于这些算法通常是为处理单个特定类型查询的单个数据流而设计的，并且大多数这些算法不能在非常高的链接速度下运行。在这个项目中，首席研究员（PI）将研究新的范例和机制，使我们能够在成千上万的高速链路和节点上执行大规模分布式数据流，并聚合、压缩和解释这些流结果，以便更好地测量和监控大型网络。这些范例和机制将被设计用于解决传统数据流算法无法处理的重要网络监控和测量问题。在初步工作的基础上，PI计划针对现有数据流算法的不足，在以下两个数据流的知识主题上进行研究。第一个研究主题是设计能够在40+ Gbps的高链路速度下运行且仍然提供高精度的数据流算法，并研究如何以一种更有效的方式实现多个数据流目标，而不是简单地将为这些目标设计的单个流算法组合在一起。第二个研究主题是设计分布式数据流算法，该算法可以在许多高速链路上的聚合流量中识别全局模式（例如，全局频繁项），而不会将流量合并到单个流中。这两个研究主题是密切相关的，构成了一个整体的努力。成功地探索网络数据流所涉及的问题将产生重大的科学和工程影响。研究结果将为我们提供更好的技术来测量、监控和管理大型高速网络，使未来的互联网基础设施更加可控、可扩展和健壮。本项目开发的结果和方法可能有助于解决许多其他网络数据流问题，并可能在数据库等其他领域有潜在的应用。更广泛的影响：该项目将吸引研究生和本科生，并为他们提供研究和学习经验，不仅在计算机网络，而且在其他领域，如统计和信息理论。这将提高他们的数学和解决问题的能力，使他们更能适应未来网络和计算的挑战。这个项目将通过引入一门关于数据流原理及其在网络和数据库中的应用的新课程来影响研究生和本科生的课程。这一努力将有助于形成研究与教育之间的牢固关系。拟议的研究将加强PI与at&t研究实验室、IBM和Telcordia研究实验室的研究人员之间正在进行的合作，促进科学发现在应用领域的应用。除了在主要会议和期刊上发表论文外，研究结果将通过以下方式广泛传播：在出席人数众多的会议上的特邀演讲和教程、组织重点讲习班、为本项目开发的数据流软件的开放来源。PI还将继续努力，积极让代表性不足的群体参与研究和教育。

Accurate traffic measurement and monitoring is critical for network management, operation, and control. With the rapid growth of the Internet, network link speeds have become faster every year to accommodate more Internet users. Measuring and monitoring the traffic on such high-speed links has become a very challenging problem. Data streaming, first introduced in the database area, has been touted as a viable solution for this problem. Data streaming is concerned with processing a long stream of data items in one pass using a small working memory in order to answer a query regarding the stream. The challenge is to use this small memory to ``remember'' as much information pertinent to the query as possible. However, traditional data streaming algorithms, designed mostly for database applications, are not suitable for future network environments, where large volumes of data flowing through numerous high-speed links need to be monitored and controlled. This is due to the fact that these algorithms are typically designed for processing a single stream of data for a single specific type of query, and most of these algorithms cannot operate at very high link speed.In this project, the principal investigator (PI) will investigate novel paradigms and mechanisms that allow us to perform large-scale distributed data streaming on tens of thousands of high-speed links and nodes, and aggregate, compress, and interpret these streaming results, for better measurement and monitoring of large networks. These paradigms and mechanisms will be designed to address the important network monitoring and measurement problems that traditional data streaming algorithms are not equipped to handle. Building on preliminary work, the PI plans to conduct research in the following two intellectual themes of data streaming, targeting the deficiency of existing data streaming algorithms. The first research theme is to design data streaming algorithms that can operate at the high link speeds of 40+ Gbps and still provide high accuracy, and to investigate how to achieve multiple data streaming goals in a much more resource-efficient way than simply combining individual streaming algorithms designed for those goals. The second research theme is to design distributed data streaming algorithms that can identify global patterns (e.g., globally frequent items) in aggregate traffic over many high-speed links, without merging the traffic into a single stream. These two research themes are closely related, constituting a holistic effort.The successful exploration of the issues involved in network data streaming will have significant scientific and engineering impact. The results will provide us with much better technology for measurement, monitoring, and management of large high-speed networks, making the future Internet infrastructure more controllable, scalable, and robust. The results and methodologies developed in this project may help solve many other network data streaming problems and may have potential applications in other fields such as databases.Broader Impact: This project will engage both graduate and undergraduate students, and offer them research and learning experience not only in computer networking, but also in other fields such as statistics and information theory. This will enhance their mathematical and problem solving skills, and make them more adaptive to the future challenges of networking and computing. This project will impact graduate and undergraduate curriculum through the introduction of a new course on principles of data streaming and its applications in networking and databases. This effort will help contribute to the formation of a strong relationship between research and education. The proposed research will strengthen the ongoing collaboration between the PI and the researchers at AT&T Labs--Research, IBM, and Telcordia Research Lab, facilitating application of scientific discoveries to the application domains. The results will be broadly disseminated through invited talks and tutorials at well-attended conferences, organization of focused workshops, and open-sourcing of data streaming software developed for this project, in addition to publication of papers in leading conferences and journals. The PI will also continue to work hard to actively engage underrepresented groups in research and education.