NeTS: Medium: From Packets to Insights: Programmable Streaming Analytics for Networks

监控我们通信网络上的互联网流量的能力对我们国家的经济繁荣和国家安全至关重要。然而，要使通信网络运行良好，网络运营商必须能够对其进行管理：他们必须能够检测、诊断和修复降低我们使用的应用程序性能的问题，并且他们必须能够检测和缓解对基础设施的攻击。*为了确保计算机网络的安全和良好运行，网络运营商需要收集测量数据以检测攻击流量、诊断性能问题、识别易损坏的设备、推动流量工程决策等。尽管网络设备提供了合理的机制来监视控制平面--负责通过网络路由分组/信息的那部分网络。但是用于监视网络流量流的工具和机制仍然是原始的(例如，用于主动测量的ping和Traceroute，用于被动测量的Netflow和sFlow)。这些测量提供了有关网络流量或条件的粗略统计信息，但它们同时提供的信息太少(因为它们模糊了有关流的重要细节，例如分组计时、队列大小和丢失率)和太多信息(因为，对于任何关于性能或安全的特定问题，运营商需要关于少数流的详细信息，而不是关于所有流的粗略信息)。该项目旨在开发适合上述每项任务的测量方法，并设计一个数据分析平台，用于查询可用于诊断和缓解网络问题的数据。这两种技术趋势使网络测量具有根本的新范式。第一个趋势是可编程网络硬件的兴起--包括可重新配置的专用集成电路(ASIC)、现场可编程门阵列(FGA)和网络处理器--这些硬件足够快、足够便宜，可以用于商用交换机，也可以用P4等与目标无关的语言进行编程。第二个趋势是出现了可伸缩的流媒体分析平台，如Spark Streaming和ApacheStorm。这些平台使基于元组流的查询表达成为可能，并有效地过滤和聚合数据。使用交换机的可编程功能，可以定义交换机导出的元组类型，甚至可以直接在数据平面中对元组执行简单计算。给定来自一个或多个交换机的输入元组流，流处理器可以计算高级查询的答案。该项目正在开发一个流分析框架，以应对这些挑战。研究人员将从现有的流分析平台开发一种具有熟悉编程范例的查询语言，他们将对其进行扩展，以支持特定于领域的原语。他们还在开发一个运行时系统，将该查询跨流处理器和数据平面中的交换机进行分区。查询将需要网络范围的聚合、迭代的“向下钻取”功能以及与外部数据源(例如，路由、应用程序标识)的连接。研究人员正在评估该平台在运行网络中出现的广泛安全和性能诊断查询的背景下的可行性和可用性。

The ability to monitor Internet traffic on our communications networks is of critical importance to our nation's economic prosperity and national security. For communications networks to run well, however, network operators must be able to manage them: they must be able to detect, diagnose, and fix problems that degrade the performance of the applications we use, and they must be able to detect and mitigate attacks against the infrastructure.  To ensure that computer networks are secure and perform well, network operators need to gather measurements to detect attack traffic, diagnose performance problems, identify flaky equipment, drive traffic-engineering decisions, and more. Although network devices provide reasonable mechanisms for monitoring the control plane -- that part of the network that is responsible for routing packets/information through the network.  Tools and mechanisms for monitoring the flow of network traffic remain primitive (e.g., ping and traceroute for active measurement, Netflow and sFlow for passive measurement). These measurements provide coarse statistics about network traffic or conditions, but they provide at once both too little information (because they obscure important details about the flows, such as packet timings, queue sizes, and loss rates) and too much information (because, for any particular question about performance or security, the operator needs detailed information about a few flows, as opposed to coarse information about all of them). This project aims to develop measurements that are "just right" for each of the above tasks and to design a data-analytics platform for querying the data that can be used to diagnose and mitigate network problems. Two technological trends enable fundamentally new paradigms for network measurement. The first trend is the rise of programmable network hardware -- including reconfigurable application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and network processors -- that are fast and inexpensive enough for use in commodity switches, and also programmable in target-independent languages like P4. The second trend is the emergence of scalable streaming analytics platforms, such as Spark Streaming and Apache Storm. These platforms make it possible to express queries based on streams of tuples and efficiently filter and aggregate the data. Using the programmable functionality from switches, one can define the types of tuples that a switch exports, and even perform simple computations over the tuples directly in the data plane. Given input tuple streams from one or more switches, the stream processor can compute the answer to a high-level query. This project is developing a streaming analytics framework that addresses these challenges. The researchers will develop a query language with familiar programming paradigms from existing streaming analytics platforms, which they will extend to support domain-specific primitives. They are also developing a runtime system that partitions this query across the stream processor and the switches in the data plane. Queries will entail network-wide aggregation, iterative "drill down" capabilities, and joins with external data sources (e.g., routing, application identification). The researchers are evaluating the feasibility and usability of this platform in the context of a wide-range of security and performance diagnosis queries that arise in operational networks.