IMR: MT: Tools for Programming Distributed Data-plane Measurements

了解跨关键网络的流量——它是由什么组成的以及它是如何变化的——对于改善现代信息服务至关重要。然而，传统上，研究人员很难开发新的工具来剖析这些流量并分析其特征，同时还要注意维护用户隐私。然而，最近，相对便宜的可编程交换机的发展使得开发诊断工具成为可能，并将它们直接放置在网络中，在流量流经的路径上。在这种情况下，新工具有可能看到所有流经的互联网流量，例如，从大学校园到更广泛的互联网，或者沿着企业广域网或数据中心。不幸的是，虽然有可能开发这样的工具，但目前这样做是一个非常困难且容易出错的过程。为了改善这种情况，研究团队将开发Lucid，这是一种新的编程语言和系统，将促进在实时可编程网络中开发、调试和部署网络测量工具的过程。研究团队将提供一个编译器，将高级Lucid程序转换为在多个地方执行的低级代码——直接在可编程交换机上执行，或者在连接到网络的服务器上支持。此外，该团队将提供一系列可重用的组件，网络测量研究人员可以将这些组件连接在一起，快速开始一个新的想法。为了教研究人员如何使用这种新语言，该团队正在为大型网络会议开发教程。总而言之，该项目将通过促进新的测量工具的开发来影响关键网络的性能、可靠性和安全性，这些工具可以发现网络优化机会、检测故障并快速识别破坏在线服务的攻击。传统的测量工具和数据集虽然非常有用，但在规模和覆盖范围方面存在重大限制。测量研究人员应该利用可编程数据平面的令人兴奋的进展来分析数据包在网络中的流量和性能。直接在数据平面（例如，网络交换机、路由器）上分析流量可以在不牺牲效率或泄露敏感用户信息的情况下进行复杂的分析，而企业网络，如大学校园，为在实践中使用这些可编程数据平面提供了极好的机会。然而，对数据平面进行编程并不容易。现有的语言，如P4，是非常低级的，有一个非常陡峭的学习曲线，并且众所周知难以使用（看起来合法的程序经常无法编译）。这个项目通过以Lucid的形式提供新的编程支持来解决这些痛点，Lucid是一种高级语言，旨在支持跨多个位置和设备类型的协作测量。更具体地说，研究团队正在开发针对英特尔Tofino可编程开关（通过P4）和软件服务器（通过eBPF）的编译器。使用这两种设备，研究人员将能够开发和部署一系列不同类型的分布式测量工具。研究小组还将开发该语言的解释器，以便在部署之前开发和调试有趣的新研究想法。研究团队开发的基础设施还将包括一套库，这些库实现了关键数据结构和实用程序，可用于网络测量和支持数据隐私。为了教社区如何使用我们的语言、库、工具和基础设施，团队将开发文档和教程。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Understanding the flow of traffic across key networks---what it is composed of and how it changes---is critical for improving modern information services. Traditionally, however, it has been difficult for researchers to develop new tools for dissecting this traffic and analyzing its characteristics, while taking care to maintain user privacy. Recently, though, the development of relatively cheap programmable switches has made it possible to develop diagnostic tools and place them directly inside the network, on the path through which traffic flows. In such a position, new tools have the potential to see all the internet traffic as it flows by, from a university campus to the broader internet, for instance, or along a corporate wide-area network or data center. Unfortunately, while it is possible to develop such tools, doing so is currently an incredibly difficult and error-prone process. To ameliorate this situation, the research team will develop Lucid, a new programming language and system that will facilitate the process of developing, debugging, and deploying network measurement tools in live programmable networks. The research team will deliver a compiler that translates high-level Lucid programs into lower-level code that execute in multiple places---directly on programmable switches, or in support, on servers connected to the network in question. In addition, the team will deliver a collection of reusable components that network measurement researchers can plug together to get started on a new idea quickly. To help teach researchers how to use the new language, the team is developing tutorials for major conferences in networking. To summarize, this project will impact the performance, reliability, and security of critical networks by facilitating the development of new measurement tools that can discover network optimization opportunities, detect failures, and rapidly recognize attacks that disrupt online services.Traditional measurement tools and datasets, while incredibly useful, have significant limitations in scale and coverage. Measurement researchers should capitalize on the exciting advances in programmable data planes to analyze Internet traffic and performance as packets traverse the network. Analyzing traffic directly in the data plane (e.g., network switches, routers) enables sophisticated analysis without sacrificing efficiency or divulging sensitive user information, and enterprise networks, such as university campuses, provide an excellent opportunity to use these programmable data planes in practice. However, programming the data plane is not easy. Existing languages, such as P4, are very low-level, have an extremely steep learning curve, and are notoriously difficult to work with (with seemingly legitimate programs often failing to compile). This project addresses these pain points by delivering new programming support in the form of Lucid, a high-level language designed to support cooperative measurement across multiple locations and device types. More specifically, the research team is developing compilers that will target both Intel Tofino programmable switches (via P4) and software servers (via eBPF). Using both kinds of devices, researchers will be able to develop and deploy a range of different kinds of distributed measurement tools. The research team will also develop an interpreter for the language so that interesting new research ideas may be developed and debugged prior to deployment. The infrastructure developed by the research team will also include a suite of libraries that implement key data structures and utilities useful in network measurement and in support of data privacy. To teach the community how to use our language, libraries, tools, and infrastructure, the team will develop documentation and tutorials.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.