Network Binarization via Contrastive Learning
Network Binarization via Contrastive Learning
复制标题
DOI：
10.48550/arxiv.2207.02970
复制
发表时间：
2022-07
期刊：
影响因子：
--
通讯作者：
Yuzhang Shang;Dan Xu;Ziliang Zong;Liqiang Nie;Yan Yan-Yan
中科院分区：
文献类型：
--
作者：
Yuzhang Shang;Dan Xu;Ziliang Zong;Liqiang Nie;Yan Yan-Yan
神经网络二值化通过将深度模型的权重和激活量化到1位来加速深度模型。然而，二值神经网络（BNNs）与全精度神经网络（FP）之间仍然存在巨大的性能差距。由于早期的研究已经减小了权值二值化带来的量化误差，激活二值化成为进一步提高精度的主要障碍。BNN具有独特而有趣的结构，其中二进制和潜在FP激活存在于相同的前向传递中（即$\text{Binarize}(\mathbf{a}_F) = \mathbf{a}_B$）。为了减轻从FP到二进制激活的二值化操作造成的信息退化，我们通过互信息（MI）最大化的视角建立了一种新的对比学习框架，同时训练bnn。MI被引入作为度量二值和FP激活之间共享信息的度量，这有助于二值化与对比学习。具体来说，通过从相同的输入样本中拉出具有二进制和FP激活的正对，以及从不同样本中推入负对（负对的数量可以呈指数级增长），极大地增强了bnn的表示能力。这有利于下游任务，不仅是分类，而且分割和深度估计等。实验结果表明，我们的方法可以作为现有最先进的二值化方法的堆积模块来实现，并且可以显着提高CIFAR-10/100和ImageNet上的性能，并且在nyad -v2上具有良好的泛化能力。
Neural network binarization accelerates deep models by quantizing their weights and activations into 1-bit. However, there is still a huge performance gap between Binary Neural Networks (BNNs) and their full-precision (FP) counterparts. As the quantization error caused by weights binarization has been reduced in earlier works, the activations binarization becomes the major obstacle for further improvement of the accuracy. BNN characterises a unique and interesting structure, where the binary and latent FP activations exist in the same forward pass (i.e., $\text{Binarize}(\mathbf{a}_F) = \mathbf{a}_B$). To mitigate the information degradation caused by the binarization operation from FP to binary activations, we establish a novel contrastive learning framework while training BNNs through the lens of Mutual Information (MI) maximization. MI is introduced as the metric to measure the information shared between binary and FP activations, which assists binarization with contrastive learning. Specifically, the representation ability of the BNNs is greatly strengthened via pulling the positive pairs with binary and FP activations from the same input samples, as well as pushing negative pairs from different samples (the number of negative pairs can be exponentially large). This benefits the downstream tasks, not only classification but also segmentation and depth estimation, etc. The experimental results show that our method can be implemented as a pile-up module on existing state-of-the-art binarization methods and can remarkably improve the performance over them on CIFAR-10/100 and ImageNet, in addition to the great generalization ability on NYUD-v2.