APNN-TC: Accelerating Arbitrary Precision Neural Networks on Ampere GPU Tensor Cores
APNN-TC: Accelerating Arbitrary Precision Neural Networks on Ampere GPU Tensor Cores
复制标题
DOI：
10.1145/3458817.3476157
复制
发表时间：
2021-06
期刊：
影响因子：
--
通讯作者：
Boyuan Feng;Yuke Wang;Tong Geng;Ang Li;Yufei Ding
中科院分区：
文献类型：
--
作者：
Boyuan Feng;Yuke Wang;Tong Geng;Ang Li;Yufei Ding
多年来，量化加速神经网络得到了广泛的研究。遗憾的是，具有不同精度(例如，1位权重和2位激活)的先前工作通常受到GPU(例如，INT1和INT4)上有限精度支持的限制。为了打破这些限制，我们引入了第一个任意精度神经网络框架(APNN-TC)1，以充分利用安培GPU张量核的量化优势。具体地说，APNN-TC首先结合了一种新的仿真算法，以支持使用int1计算基元和XOR/AND布尔运算的任意短位宽计算。其次，APNN-TC集成了任意精度层设计，以新颖的批处理策略和专门的存储结构将我们的仿真算法高效地映射到张量核上。第三，APNN-TC采用了一种新颖的任意精度神经网络设计，以最大限度地减少跨层内存访问，并进一步提高性能。大量的测试结果表明，APNN-TC在截断类核和各种神经网络模型(如ResNet和VGG)上都能获得显著的加速比。
Over the years, accelerating neural networks with quantization has been widely studied. Unfortunately, prior efforts with diverse precisions (e.g., 1-bit weights and 2-bit activations) are usually restricted by limited precision support on GPUs (e.g., int1 and int4). To break such restrictions, we introduce the first Arbitrary Precision Neural Network framework (APNN-TC)1 to fully exploit quantization benefits on Ampere GPU Tensor Cores. Specifically, APNN-TC first incorporates a novel emulation algorithm to support arbitrary short bit-width computation with int1 compute primitives and XOR/AND Boolean operations. Second, APNN-TC integrates arbitrary precision layer designs to efficiently map our emulation algorithm to Tensor Cores with novel batching strategies and specialized memory organization. Third, APNN-TC embodies a novel arbitrary precision NN design to minimize memory access across layers and further improve performance. Extensive evaluations show that APNN-TC can achieve significant speedup over CUT-LASS kernels and various NN models, such as ResNet and VGG.