Efficient User-Level Storage Disaggregation for Deep Learning
Efficient User-Level Storage Disaggregation for Deep Learning
复制标题
DOI：
10.1109/cluster.2019.8891023
复制
发表时间：
2019-09
期刊：
影响因子：
--
通讯作者：
Yue Zhu;Weikuan Yu;Bing Jiao;K. Mohror;A. Moody;Fahim Chowdhury
中科院分区：
文献类型：
--
作者：
Yue Zhu;Weikuan Yu;Bing Jiao;K. Mohror;A. Moody;Fahim Chowdhury
在大规模高性能计算（HPC）系统上，为应用程序提供聚合资源，以在短时间内满足其峰值需求。这将导致资源利用不足，因为在执行过程中应用程序的需求变化很大。对于运行在领先的高性能计算系统上的深度学习应用程序来说，这个问题尤其明显，这些系统具有大量闪存或非易失性存储器（NVM）设备形式的突发缓冲池。在本文中，我们研究了深度神经网络的I/O模式，并揭示了它们对随机加载许多小样本以成功训练的迫切需求。我们设计了一个专门的深度学习文件系统（DLFS），它提供了一组精简的api。特别地，我们通过基于内存树的样本目录设计了DLFS的元数据管理，并通过用户级SPDK协议设计了DLFS的文件服务，该协议可以分解NVM Express （NVMe）设备的能力以并行训练任务。我们的实验结果表明，与基于内核的Ext4文件系统相比，DLFS可以显著提高NVMe在Fabric上的深度神经网络训练吞吐量。此外，DLFS以非常低的CPU利用率实现了高效的用户级存储分解。
On large-scale high performance computing (HPC) systems, applications are provisioned with aggregated resources to meet their peak demands for brief periods. This results in resource underutilization because application requirements vary a lot during execution. This problem is particularly pronounced for deep learning applications that are running on leadership HPC systems with a large pool of burst buffers in the form of flash or non-volatile memory (NVM) devices. In this paper, we examine the I/O patterns of deep neural networks and reveal their critical need of loading many small samples randomly for successful training. We have designed a specialized Deep Learning File System (DLFS) that provides a thin set of APIs. Particularly, we design the metadata management of DLFS through an in-memory tree-based sample directory and its file services through the user-level SPDK protocol that can disaggregate the capabilities of NVM Express (NVMe) devices to parallel training tasks. Our experimental results show that DLFS can dramatically improve the throughput of training for deep neural networks on NVMe over Fabric, compared with the kernel-based Ext4 file system. Furthermore, DLFS achieves efficient user-level storage disaggregation with very little CPU utilization.