Understanding the Micro-Behaviors of Hardware Offloaded Network Stacks with Lumina
Understanding the Micro-Behaviors of Hardware Offloaded Network Stacks with Lumina
复制标题
DOI：
10.1145/3603269.3604837
复制
发表时间：
2023-09
期刊：
影响因子：
--
通讯作者：
Zhuolong Yu;Bo-Wei Su;Wei Bai;S. Raindel;V. Braverman;Xin Jin
中科院分区：
文献类型：
--
作者：
Zhuolong Yu;Bo-Wei Su;Wei Bai;S. Raindel;V. Braverman;Xin Jin
为了满足高吞吐量、超低延迟和低CPU开销的需求，现代数据中心广泛采用硬件分流网络堆栈。要充分利用其卓越的性能，用户需要深入了解他们的行为。尽管在测试软件网络协议栈方面做了很多努力，但硬件网络协议栈由于其绕过内核的性质和高性能，给测试工具带来了独特的挑战。本文介绍了一种测试硬件网络协议栈正确性和性能的工具Lumina。Lumina利用网络可编程性以线速模拟各种网络场景。通过用户友好的界面，Lumina使开发人员能够注入确定性事件，从而促进精确和可重复性测试的开发。考虑到可编程网络设备的有限资源和灵活性，我们将所有数据包镜像到专用服务器，并将其转储以供离线分析。我们利用Lumina测试来自NVIDIA和英特尔的四个RDMA网卡，并找出可能会显著降低性能或误导网络运营的错误。Lumina还使我们能够捕获公共文档和规范中缺失或未明确描述的意外微观行为。供应商已经确认了我们发现的关键错误，并将在未来的版本中包括错误修复。
Hardware offloaded network stacks are widely adopted in modern datacenters to meet the demand for high throughput, ultra-low latency and low CPU overhead. To fully leverage their exceptional performance, users need to have a deep understanding of their behaviors. Despite many efforts on testing software network stacks, hardware network stacks impose unique challenges to testing tools due to their kernel bypass nature and high performance. In this paper, we present Lumina, a tool to test the correctness and performance of hardware network stacks. Lumina leverages network programmability to emulate various network scenarios at line rate. With user-friendly interfaces, Lumina enables developers to inject deterministic events, thus facilitating the development of precise and reproducible tests. Given the limited resource and flexibility of programmable network devices, we mirror all the packets to dedicated servers and dump them for offline analysis. We leverage Lumina to test four RDMA NICs from NVIDIA and Intel, and identify bugs that can significantly degrade performance or mislead network operations. Lumina also enables us to capture unexpected micro-behaviors which are missing or not clearly described in public documents and specifications. Vendors have confirmed the critical bugs we discovered and will include bug fixes in future releases.