Accelerating Tensor Swapping in GPUs With Self-Tuning Compression
Accelerating Tensor Swapping in GPUs With Self-Tuning Compression
复制标题
DOI：
10.1109/tpds.2022.3193867
复制
发表时间：
2022-12
影响因子：
5.3
通讯作者：
Ping Chen;Shuibing He;Xuechen Zhang;Shuaiben Chen;Peiyi Hong;Yanlong Yin;Xian-He Sun
中科院分区：
文献类型：
--
作者：
Ping Chen;Shuibing He;Xuechen Zhang;Shuaiben Chen;Peiyi Hong;Yanlong Yin;Xian-He Sun
CPU和GPU之间的数据交换被广泛用于解决训练深度神经网络（DNN）时的GPU内存短缺问题，因为DNN需要比GPU更大的内存。当数据交换的延迟长于DNN计算的延迟时，数据交换可能成为瓶颈。GPU中的张量压缩可以减少数据交换时间。然而，在GPU的虚拟内存中压缩张量的现有工作有三个主要问题：缺乏可移植性，因为它的实现需要额外的（解）压缩单元在内存控制器中，次优压缩性能为不同的张量压缩比和大小，和穷人的适应密集张量，因为他们只关注稀疏张量。我们提出了一个自调优张量压缩框架，名为CSWap+，用于改善GPU的虚拟内存管理。它直接使用GPU进行压缩（解压缩），因此具有很高的可移植性，并且对GPU架构特性的依赖性最小。此外，它只对被认为是具有成本效益的张量应用压缩，考虑到它们的压缩比，大小和压缩算法在运行时的特性。最后，为了适应具有密集张量的DNN模型，它还支持对密集张量进行具有成本效益的有损压缩，几乎没有模型训练精度下降。我们通过六个代表性的记忆密集型DNN模型进行实验。与vDNN相比，CSwap+在NVIDIA V100 GPU上分别将具有稀疏和密集张量的DNN模型的张量交换延迟降低了50.9%和46.1%。
Data swapping between CPUs and GPUs is widely used to address the GPU memory shortage issue when training deep neural networks (DNNs) requiring a larger amount of memory than that a GPU may have. Data swapping may become a bottleneck when its latency is longer than the latency of DNN computations. Tensor compression in GPUs can reduce the data swapping time. However, existing works on compressing tensors in the virtual memory of GPUs have three major issues: lack of portability because its implementation requires additional (de)compression units in memory controllers, sub-optimal compression performance for varying tensor compression ratios and sizes, and poor adaptation to dense tensors because they only focus on sparse tensors. We propose a self-tuning tensor compression framework, named CSwap+, for improving the virtual memory management of GPUs. It uses GPUs for (de)compression directly and thus has high portability and is minimally dependent on GPU architecture features. Furthermore, it only applies compression on tensors that are deemed to be cost-effective considering their compression ratio, size, and the characteristics of compression algorithms at runtime. Finally, to adapt to DNN models with dense tensors, it also supports cost-effective lossy compression for dense tensors with nearly no model training accuracy degradation. We conduct the experiments through six representative memory-intensive DNN models. Compared to vDNN, CSwap+ reduces tensor swapping latency by up to 50.9% and 46.1% with NVIDIA V100 GPU, for DNN models with sparse and dense tensors, respectively.