Craft Distillation: Layer-wise Convolutional Neural Network Distillation
Craft Distillation: Layer-wise Convolutional Neural Network Distillation
复制标题
DOI：
10.1109/cscloud-edgecom49738.2020.00051
复制
发表时间：
2020-08
期刊：
影响因子：
--
通讯作者：
Cody Blakeney;Xiaomin Li;Yan Yan-Yan;Ziliang Zong
中科院分区：
文献类型：
--
作者：
Cody Blakeney;Xiaomin Li;Yan Yan-Yan;Ziliang Zong
卷积神经网络（CNN）在解决许多具有挑战性的计算机视觉任务方面取得了巨大成功。然而，CNN对计算能力、存储空间和功率容量的要求非常高。这限制了它们对云的使用，并阻止它们部署在资源和功率受限的边缘设备上。为了解决这个问题，我们提出了工艺蒸馏，一种新的模型压缩方法，利用依赖可分离卷积和知识蒸馏来显着减少高度复杂模型的大小。工艺蒸馏与现有的模型压缩技术相比有三个优点。首先，它不需要事先就如何设计一个好的“学生模型”进行有效的知识蒸馏的经验。其次，它不需要专门的硬件支持（例如ASIC或FPGA）。第三，它与现有的模型压缩技术兼容，并且可以与修剪和量化一起使用，以进一步减少权重存储和算术运算。我们的实验结果表明，通过适当的层块替换设计和替换策略，工艺蒸馏降低了VGG16的计算成本的74.6%相比，原来的密集模型的精度可以忽略不计的影响。
Convolutional neural networks (CNNs) have achieved tremendous success in solving many challenging computer vision tasks. However, CNNs are extremely demanding for computation capability, memory space, and power capacity. This limits their usage to the cloud and prevents them from being deployed on edge devices with constrained resources and power. To tackle this problem, we propose craft distillation, a novel model compression approach that leverages both depthwise separable convolutions and knowledge distillation to significantly reduce the size of a highly complex model. Craft distillation has three advantages over existing model compression techniques. First, it does not require prior experiences on how to design a good “student model” for effective knowledge distillation. Second, it does not require specialized hardware support (e.g. ASIC or FPGA). Third, it is compatible with existing model compression techniques and can be used with pruning and quantization together to further reduce weight storage and arithmetic operations. Our experimental results show that with proper layer block replacement design and replacement strategy, craft distillation reduces the computational cost of VGG16 by 74.6% compared to the original dense models with negligible influence on accuracy.