Build a Compact Binary Neural Network through Bit-level Sensitivity and Data Pruning
Build a Compact Binary Neural Network through Bit-level Sensitivity and Data Pruning
复制标题
DOI：
10.1016/j.neucom.2020.02.012
复制
发表时间：
2018-02
期刊：
影响因子：
6
通讯作者：
Yixing Li;Fengbo Ren
中科院分区：
文献类型：
--
作者：
Yixing Li;Fengbo Ren
由于高计算复杂度和内存存储要求，很难在嵌入式设备上直接部署全精度卷积神经网络（CNN）。嵌入式设备资源有限、能耗高，需要硬件友好的设计。新兴的解决方案被用于神经网络压缩，例如，二值/三值权值网络、剪枝网络和量化网络。其中，二进制神经网络（BNN）被认为是最硬件友好的框架，由于其小的网络规模和低计算复杂度。没有现有的工作进一步缩小BNN的大小。在这项工作中，我们探讨了冗余的BNN，并建立了一个紧凑的BNN（CBNN）的基础上的位级敏感性分析和位级数据修剪。输入数据被转换为高维位分片格式。在后训练阶段，我们分析了不同比特片对准确率的影响。通过修剪冗余的输入位片和缩小网络的大小，我们能够建立一个更紧凑的BNN。我们的结果表明，我们可以进一步缩小BNN的网络规模，最高可达3.9倍，准确率下降不超过1%。与基线BNN和全精度BNN相比，实际运行时间分别减少了2倍和9.9倍。
Due to the high computational complexity and memory storage requirement, it is hard to directly deploy a full-precision convolutional neural network (CNN) on embedded devices. The hardware-friendly designs are needed for resource-limited and energy-constrained embedded devices. Emerging solutions are adopted for the neural network compression, e.g., binary/ternary weight network, pruned network and quantized network. Among them, binary neural network (BNN) is believed to be the most hardware-friendly framework due to its small network size and low computational complexity. No existing work has further shrunk the size of BNN. In this work, we explore the redundancy in BNN and build a compact BNN (CBNN) based on the bit-level sensitivity analysis and bit-level data pruning. The input data is converted to a high dimensional bit-sliced format. In the post-training stage, we analyze the impact of different bit slices to the accuracy. By pruning the redundant input bit slices and shrinking the network size, we are able to build a more compact BNN. Our result shows that we can further scale down the network size of the BNN up to 3.9x with no more than 1% accuracy drop. The actual runtime can be reduced up to 2x and 9.9x compared with the baseline BNN and its full-precision counterpart, respectively.