Revisiting the Classics: Online RL in the Programmable Dataplane
Revisiting the Classics: Online RL in the Programmable Dataplane
复制标题
DOI：
10.1109/noms54207.2022.9789930
复制
发表时间：
2022-04
期刊：
影响因子：
--
通讯作者：
Kyle A. Simpson;D. Pezaros
中科院分区：
文献类型：
--
作者：
Kyle A. Simpson;D. Pezaros
数据驱动的网络正变得越来越强大，研究也越来越广泛，部分原因是深度强化学习（DRL）算法的有效性。然而，DRL推理和学习的复杂性迫使这些任务从数据平面推到主机，损害了延迟敏感的应用程序。在线学习这样的政策不能发生在dataplane，尽管是有用的技术时，问题的发展或难以model.We介绍OPaL路径学习的第一个工作，使在线强化学习的dataplane。OPaL通过返回经典的RL技术（避免神经网络），使在线学习在受限的SmartNIC硬件中成为可能。我们的设计允许弱但高度并行的SmartNIC NPU与商品x86主机竞争，尽管功能更少，内核更慢。与主机相比，我们将99.99th tail推理时间减少了21倍，达到34 µs，并将实际策略设计的在线吞吐量提高了9.9倍。In-NIC执行消除了PCIe传输，我们的异步计算模型确保对共同托管的P4数据平面承载的流量的影响最小。OPaL的设计可在编译时扩展额外的资源，以改善决策延迟和吞吐量，并且与重新安装设备固件相比，可在运行时快速重新配置。
Data-driven networking is becoming more capable and widely researched, partly driven by the efficacy of Deep Reinforcement Learning (DRL) algorithms. Yet the complexity of both DRL inference and learning force these tasks to be pushed away from the dataplane to hosts, harming latency-sensitive applications. Online learning of such policies cannot occur in the dataplane, despite being useful techniques when problems evolve or are hard to model.We present OPaL—On Path Learning—the first work to bring online reinforcement learning to the dataplane. OPaL makes online learning possible in constrained SmartNIC hardware by returning to classical RL techniques—avoiding neural networks. Our design allows weak yet highly parallel SmartNIC NPUs to be competitive against commodity x86 hosts, despite having fewer features and slower cores. Compared to hosts, we achieve a 21 × reduction in 99.99th tail inference times to 34 µs, and 9.9 × improvement in online throughput for real-world policy designs. In-NIC execution eliminates PCIe transfers, and our asynchronous compute model ensures minimal impact on traffic carried by a co-hosted P4 dataplane. OPaL’s design scales with additional resources at compile-time to improve upon both decision latency and throughput, and is quickly reconfigurable at runtime compared to reinstalling device firmware.