AutoQ: Automated Kernel-Wise Neural Network Quantization
AutoQ: Automated Kernel-Wise Neural Network Quantization
复制标题
DOI：
--
复制
发表时间：
2019-02
期刊：
影响因子：
--
通讯作者：
Qian Lou;Feng Guo;Lantao Liu;Minje Kim;Lei Jiang
中科院分区：
文献类型：
--
作者：
Qian Lou;Feng Guo;Lantao Liu;Minje Kim;Lei Jiang
网络量化是使卷积神经网络(CNN)能够在低功耗移动设备上部署的最硬件友好的技术之一。最近的网络量化技术独立地量化卷积层中的每个加权核，以获得更高的推理精度，因为层中的加权核表现出不同的方差，因此具有不同的冗余量。量化比特宽度或比特数(QBN)直接决定推理的精度、延迟、能量和硬件开销。为了有效地减少冗余度，加速CNN推理，需要用不同的QBN对不同的权核进行量化。然而，以往的工作只使用一个QBN来量化每个卷积层或整个CNN，这是因为为每个权核搜索一个QBN的设计空间太大。基于内核的QBN搜索的手工启发式方法是如此复杂，以至于领域专家只能获得次优结果。即使是基于深度强化学习(DRL)和深度确定性策略梯度(DDPG)的代理也很难找到能够达到合理推理精度的核级QBN结构。在本文中，我们提出了一种基于分层DRL的核式网络量化技术AutoQ，它自动为每个权核搜索一个QBN，并为每个激活层选择另一个QBN。与最新的基于DRL的量化模型相比，AutoQ量化的相同模型平均减少了54.06%的推理延迟，降低了50.69%的推理能耗，同时达到了相同的推理精度。
Network quantization is one of the most hardware friendly techniques to enable the deployment of convolutional neural networks (CNNs) on low-power mobile devices. Recent network quantization techniques quantize each weight kernel in a convolutional layer independently for higher inference accuracy, since the weight kernels in a layer exhibit different variances and hence have different amounts of redundancy. The quantization bitwidth or bit number (QBN) directly decides the inference accuracy, latency, energy and hardware overhead. To effectively reduce the redundancy and accelerate CNN inferences, various weight kernels should be quantized with different QBNs. However, prior works use only one QBN to quantize each convolutional layer or the entire CNN, because the design space of searching a QBN for each weight kernel is too large. The hand-crafted heuristic of the kernel-wise QBN search is so sophisticated that domain experts can obtain only sub-optimal results. It is difficult for even deep reinforcement learning (DRL) Deep Deterministic Policy Gradient (DDPG)-based agents to find a kernel-wise QBN configuration that can achieve reasonable inference accuracy. In this paper, we propose a hierarchical-DRL-based kernel-wise network quantization technique, AutoQ, to automatically search a QBN for each weight kernel, and choose another QBN for each activation layer. Compared to the models quantized by the state-of-the-art DRL-based schemes, on average, the same models quantized by AutoQ reduce the inference latency by 54.06\%, and decrease the inference energy consumption by 50.69\%, while achieving the same inference accuracy.