Parallel Blockwise Knowledge Distillation for Deep Neural Network Compression
Parallel Blockwise Knowledge Distillation for Deep Neural Network Compression
复制标题
DOI：
10.1109/tpds.2020.3047003
复制
发表时间：
2020-12
影响因子：
5.3
通讯作者：
Cody Blakeney;Xiaomin Li;Yan Yan-Yan;Ziliang Zong
中科院分区：
文献类型：
--
作者：
Cody Blakeney;Xiaomin Li;Yan Yan-Yan;Ziliang Zong
深度神经网络（DNN）在解决当今自然语言处理、语音识别和计算机视觉中许多具有挑战性的人工智能任务方面取得了巨大成功。然而，DNN通常是计算密集型的，内存需求量大，耗电量大，这大大限制了它们在资源受限的平台上的使用。因此，各种压缩技术（例如，量化、修剪和知识蒸馏）来减小DNN的大小和功耗。知识蒸馏是一种压缩技术，可以有效地减少高度复杂的DNN的大小。但由于训练时间较长，没有被广泛采用。在这篇文章中，我们提出了一种新的并行分块蒸馏算法来加速复杂DNN的蒸馏过程。我们的算法利用本地信息进行独立的分块蒸馏，利用dependency可分离层作为有效的替换块架构，并适当地解决限制因素（例如，依赖性、同步和负载平衡）。在AMD服务器上运行四个Geforce RTX 2080Ti GPU的实验结果表明，我们的算法可以实现3倍的加速和19%的节能VGG蒸馏，和3.5倍的加速和29%的节能ResNet蒸馏，两者都可以忽略不计的准确性损失。当在分布式集群中使用四个RTX6000 GPU时，ResNet蒸馏的加速比可以进一步提高到3.87。
Deep neural networks (DNNs) have been extremely successful in solving many challenging AI tasks in natural language processing, speech recognition, and computer vision nowadays. However, DNNs are typically computation intensive, memory demanding, and power hungry, which significantly limits their usage on platforms with constrained resources. Therefore, a variety of compression techniques (e.g., quantization, pruning, and knowledge distillation) have been proposed to reduce the size and power consumption of DNNs. Blockwise knowledge distillation is one of the compression techniques that can effectively reduce the size of a highly complex DNN. However, it is not widely adopted due to its long training time. In this article, we propose a novel parallel blockwise distillation algorithm to accelerate the distillation process of sophisticated DNNs. Our algorithm leverages local information to conduct independent blockwise distillation, utilizes depthwise separable layers as the efficient replacement block architecture, and properly addresses limiting factors (e.g., dependency, synchronization, and load balancing) that affect parallelism. The experimental results running on an AMD server with four Geforce RTX 2080Ti GPUs show that our algorithm can achieve 3x speedup plus 19 percent energy savings on VGG distillation, and 3.5x speedup plus 29 percent energy savings on ResNet distillation, both with negligible accuracy loss. The speedup of ResNet distillation can be further improved to 3.87 when using four RTX6000 GPUs in a distributed cluster.