Taming Unstructured Sparsity on GPUs via Latency-Aware Optimization
Taming Unstructured Sparsity on GPUs via Latency-Aware Optimization
复制标题
DOI：
10.1109/dac18072.2020.9218644
复制
发表时间：
2020-07
期刊：
影响因子：
--
通讯作者：
Maohua Zhu;Yuan Xie
中科院分区：
文献类型：
--
作者：
Maohua Zhu;Yuan Xie
神经网络具有很高的参数冗余度，因此剪枝方法可以在不损失精度的情况下获得较高的压缩比。然而，非结构化剪枝方法产生的高度稀疏性很难有效地映射到图形处理单元(GPU)上，因为它的解码开销和工作量不平衡。随着张量内核的引入，最新的GPU为密集的神经网络实现了更高的吞吐量。这使得非结构化神经网络的性能无法超过密集神经网络，因为它们目前没有得到张量核心的支持。为了解决这个问题，以前的工作建议通过结构化剪枝来提高稀疏网络在GPU上的性能。然而，这种结构化的剪枝方法必须牺牲很大一部分稀疏性来保持模型的准确性，这限制了硬件的加速比。在本文中，我们观察到张量核也能够有效地计算非结构化稀疏神经网络。为了实现这一目标，我们首先提出了Extensor，这是一组具有可变输入矩阵瓦片大小的稀疏张量核心指令。可变瓦片大小允许通过混合不同类型的扩展器指令来实现矩阵乘法。在给定操作数稀疏权重矩阵的情况下，我们建立了一个性能模型来估计扩展指令的延迟。基于该模型，我们提出了一种启发式算法来寻找基于扩展器的内核的最优指令序列，以在GPU上获得最佳的性能。实验结果表明，该方法的性能比目前最先进的稀疏张量核设计方法提高了36%。
Neural Networks (NNs) exhibit high redundancy in their parameters so that pruning methods can achieve high compression ratio without accuracy loss. However, the very high sparsity produced by unstructured pruning methods is difficult to be efficiently mapped onto Graphics Processing Units (GPUs) because of its decoding overhead and workload imbalance. With the introduction of Tensor Core, the latest GPUs achieve even higher throughput for the dense neural networks. This makes unstructured neural networks fail to outperform their dense counterparts because they are not currently supported by Tensor Core. To tackle this problem, prior work suggests structured pruning to improve the performance of sparse NNs on GPUs. However, such structured pruning methods have to sacrifice a significant part of sparsity to retain the model accuracy, which limits the speedup on the hardware. In this paper, we observe that the Tensor Core is also able to compute unstructured sparse NNs efficiently. To achieve this goal, we first propose ExTensor, a set of sparse Tensor Core instructions with a variable input matrix tile size. The variable tile size allows a matrix multiplication to be implemented by mixing different types of ExTensor instructions. We build a performance model to estimate the latency of an ExTensor instruction given an operand sparse weight matrix. Based on this model, we propose a heuristic algorithm to find the optimal sequence of the instructions for an ExTensor based kernel to achieve the best performance on the GPU. Experimental results demonstrate that our approach achieves 36% better performance than the state-of-the-art sparse Tensor Core design.