NVMe-oAF: Towards Adaptive NVMe-oF for IO-Intensive Workloads on HPC Cloud
NVMe-oAF: Towards Adaptive NVMe-oF for IO-Intensive Workloads on HPC Cloud
复制标题
DOI：
10.1145/3502181.3531476
复制
发表时间：
2022-06
期刊：
影响因子：
--
通讯作者：
Arjun Kashyap;Xiaoyi Lu
中科院分区：
文献类型：
--
作者：
Arjun Kashyap;Xiaoyi Lu
在容器或虚拟机内运行的应用程序传统上使用TCP/IP在HPC云和数据中心中进行通信。TCP/IP路径通常会成为在分散存储设置中执行基于NVMe-over-Fabrics（NVMe-oF）的I/O操作的应用程序的主要性能瓶颈。我们提出了一种自适应通信通道，称为NVMe-over-Adaptive-Fabric（NVMe-oAF），应用程序可以利用它来消除TCP/IP上远程I/O请求所带来的高延迟和低带宽。NVMe-oAF使用位置感知沿着以及优化的共享内存和TCP/IP路径来加速I/O密集型应用程序。结构的自适应性源于自适应地选择共享内存或TCP通道并进一步对所选通道应用优化的能力。为了评估NVMe-oAF，我们将英特尔的SPDK库与我们的设计进行了联合设计，并在基于TCP/IP的商用以太网网络（例如，10 Gbps、25 Gbps和100 Gbps）。通过避免HPC云环境中繁琐的RDMA管理，我们实现了与NVMe-over-RDMA相似（有时甚至更好）的性能。最后，我们还与H5 bench共同设计了NVMe-oAF，以展示它为HDF 5应用带来的好处。我们的评估表明，与网络文件系统（NFS）相比，带宽提高了7倍。
Applications running inside containers or virtual machines, traditionally use TCP/IP for communication in HPC clouds and data centers. The TCP/IP path usually becomes a major performance bottleneck for applications performing NVMe-over-Fabrics (NVMe-oF) based I/O operations in disaggregated storage settings. We propose an adaptive communication channel, called NVMe-over-Adaptive-Fabric (NVMe-oAF), that applications could leverage to eliminate the high-latency and low-bandwidth incurred by remote I/O requests over TCP/IP. NVMe-oAF accelerates I/O intensive applications using locality awareness along with optimized shared memory and TCP/IP paths. The adaptiveness of the fabric stems from the ability to adaptively select shared memory or TCP channel and further applying optimizations for the chosen channel. To evaluate NVMe-oAF, we co-design Intel's SPDK library with our designs and show up to 7.1x bandwidth improvement and up to 4.2x latency reduction for various workloads over commodity TCP/IP-based Ethernet networks (e.g., 10Gbps, 25Gbps, and 100Gbps). We achieve similar (or sometimes better) performance when compared to NVMe-over-RDMA by avoiding the cumbersome management of RDMA in HPC cloud environments. Finally, we also co-design NVMe-oAF with H5bench to showcase the benefit it brings to HDF5 applications. Our evaluation indicates up to a 7x bandwidth improvement when compared with the network file system (NFS).