PokeBNN: A Binary Pursuit of Lightweight Accuracy
PokeBNN: A Binary Pursuit of Lightweight Accuracy
复制标题
DOI：
10.1109/cvpr52688.2022.01215
复制
发表时间：
2021-11
期刊：
影响因子：
--
通讯作者：
Yichi Zhang;Zhiru Zhang;Lukasz Lew
中科院分区：
文献类型：
--
作者：
Yichi Zhang;Zhiru Zhang;Lukasz Lew
Top-1 Imagenet的优化促进了可能在推理环境中不切实际的巨大网络。二进制神经网络（BNN）有可能显着降低计算强度，但现有模型的质量低。为了克服这种缺陷，我们提出了Poke-Conv，这是一个二元卷积块，通过添加多个残留路径和调整激活函数等技术来提高BNN的质量。我们将其应用于Resnet-50，并优化了Resnet的初始卷积层，该卷积层很难二进制。我们命名由此产生的网络家族pokebnn11poke/pnki/的发音与口袋相似。 Pokeconv，Pokebnn和Pokemon分别是口袋卷积，袖珍二进制神经网络和口袋怪物的缩写。选择这些技术以在Top-1准确性和网络的成本方面取得有利的提高。为了使成本的联合优化及其准确性，我们定义了算术计算工作（ACE），这是一种用于量化和二进制网络的硬件和能源启发的成本度量。我们还确定需要优化控制二进制梯度近似的不足探索的高参数。我们在TOP-1的准确性上建立了一个新的，强大的最先进（SOTA），以及常用的CPU64成本，ACE成本和网络尺寸指标。 BNNS先前的SOTA ReactNet-Adam [33]以7.9 ACE的成绩达到了70.5％的TOP-1精度。一小部分Pokebnn以2.6 ACE的成本达到70.5％的TOP-1，成本降低了3倍。较大的Pokebnn以7.8 ACE的优势获得75.6％的TOP-1，准确性提高了5％，而不会增加成本。 JAX/Flax [6，18]中的POKEBNN实现，并开源重新制作指令。22Source代码和复制指令在AQT repos-tority中可用：github.com/google/aqt。
Optimization of Top-1 ImageNet promotes enormous networks that may be impractical in inference settings. Binary neural networks (BNNs) have the potential to significantly lower the compute intensity but existing models suffer from low quality. To overcome this deficiency, we propose Poke- Conv, a binary convolution block which improves quality of BNNs by techniques such as adding multiple residual paths, and tuning the activation function. We apply it to ResNet-50 and optimize ResNet's initial convolutional layer which is hard to binarize. We name the resulting network family PokeBNN11Poke/pnki/is pronounced similarly to pocket. PokeConv, PokeBNN, and Pokemon are abbreviations of Pocket Convolution, Pocket Binary Neural Network, and Pocket Monster, respectively.. These techniques are chosen to yield favorable improvements in both top-1 accuracy and the network's cost. In order to enable joint optimization of the cost together with accuracy, we define arithmetic computation effort (ACE), a hardware- and energy-inspired cost metric for quantized and binarized networks. We also identify a need to optimize an under-explored hyper-parameter controlling the binarization gradient approximation. We establish a new, strong state-of-the-art (SOTA) on top-1 accuracy together with commonly-used CPU64 cost, ACE cost and network size metrics. ReActNet-Adam [33], the previous SOTA in BNNs, achieved a 70.5% top-1 accuracy with 7.9 ACE. A small variant of PokeBNN achieves 70.5% top-1 with 2.6 ACE, more than 3x reduction in cost; a larger PokeBNN achieves 75.6% top-1 with 7.8 ACE, more than 5% improvement in accuracy without increasing the cost. PokeBNN implementation in JAX/Flax [6, 18] and re-production instructions are open sourced.22Source code and reproduction instructions are available in AQT repos-itory: github.com/google/aqt.