uNVMe-TCP: A User Space Approach to Optimizing NVMe over Fabrics TCP Transport
uNVMe-TCP: A User Space Approach to Optimizing NVMe over Fabrics TCP Transport
复制标题
uNVMe-TCP：优化 NVMe over Fabrics TCP 传输的用户空间方法
DOI：
--
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
Karol Latecki
中科院分区：
文献类型：
--
作者：
Ziye Yang;Qun Wan;Gang Cao;Karol Latecki
最近，NVM Express®发布了NVMe over Fabric（NVMe-oF）的新TCP传输规范。有两种实现方式，即，一个在内核空间，另一个在用户空间。内核中的实现（例如，Linux内核）是可行的，但存在性能、灵活性和稳定性等几个缺点。在本文中，我们想介绍uNVMe-TCP，它遵循规范，并在用户空间中提供NVMe/TCP传输，具有更好的性能和使用体验。我们选择在用户空间中进行优化，因为通过不同的内核模块在内核空间中优化整个NVMe I/O堆栈非常困难，并且优化可能会影响用户空间中的其他应用程序。uNVMe-TCP的思想是在TCP传输上优化整个NVMe I/O堆栈，即，利用无锁的用户空间NVMe I/O堆栈和可配置的网络I/O堆栈（可以支持内核和用户空间TCP堆栈）。目前uNVMe-TCP提供了目标端和发起端的解决方案，并且可以针对Linux内核解决方案进行测试，具有良好的互操作性。此外，还通过实验验证了uNVMe-TCP的性能。与内核解决方案相比，uNVMe-TCP在FIO基准测试中显示出平均15%至30%的延迟改善。uNVMe-TCP的每CPU核心性能是有希望的，即，随着连接数的增加，平均为内核的2.2倍。此外，uNVMe-TCP在CPU方面也是可扩展的。
Recently, NVM Express® has released the new TCP transport specification for NVMe over fabrics (NVMe-oF). And there are two kinds of implementations, i.e., one in kernel space and the other in user space. The implementation in the kernel (e.g., Linux kernel) is feasible, but there are several drawbacks such as performance, flexibility, and stability. In this paper, we would like to introduce uNVMe-TCP, which follows the specification and provides the NVMe/TCP transport in user space with improved performance and usage experience. We choose the optimization in user space since it is very difficult to optimize the whole NVMe I/O stack in kernel space through different kernel modules, and the optimization may affect other applications in user space. The idea of uNVMe-TCP is to optimize the whole NVMe I/O stack on TCP transport, i.e., leveraging the lock-free user space NVMe I/O stack and configurable network I/O stack (both kernel and user space TCP stack can be supported). Currently uNVMe-TCP provides the solution on both target and initiator side, and it can be tested against Linux kernel solution with good interoperability. Besides, some experiments are conducted to demonstrate the performance of uNVMe-TCP. Compared with the kernel solution, uNVMe-TCP shows 15% to 30% latency improvement on average with FIO benchmark. And the per CPU core performance of uNVMe-TCP is promising, i.e., it is 2.2 times of the kernel on average with the increasing number of connections. Furthermore, uNVMe-TCP is also scalable in CPU aspect.