Convolutional Neural Networks using Logarithmic Data Representation
Convolutional Neural Networks using Logarithmic Data Representation
复制标题
DOI：
--
复制
发表时间：
2016-03
期刊：
影响因子：
--
通讯作者：
D. Miyashita;Edward H. Lee;B. Murmann
中科院分区：
文献类型：
--
作者：
D. Miyashita;Edward H. Lee;B. Murmann
卷积神经网络的最新进展已经考虑了模型复杂性和硬件效率，以实现在嵌入式系统和移动的设备上的部署。例如，现在众所周知，深度网络的算术运算可以被编码到8位定点，而不会显著降低性能。然而，精度进一步降低到低至3位定点会导致性能的显著损失。在本文中，我们提出了一种新的数据表示，使国家的最先进的网络被编码到3位，在分类性能的损失可以忽略不计。为了实现这一点，我们利用了这样一个事实，即训练网络中的权重和激活自然具有非均匀分布。使用非均匀的以2为底的对数表示来编码权重、传递激活信息和执行点积，使网络能够1）在相同分辨率下实现比定点更高的分类精度，2）消除庞大的数字乘法器。最后，我们提出了一个端到端的训练过程，使用5位的日志表示，这比5位的线性测试精度更高。
Recent advances in convolutional neural networks have considered model complexity and hardware efficiency to enable deployment onto embedded systems and mobile devices. For example, it is now well-known that the arithmetic operations of deep networks can be encoded down to 8-bit fixed-point without significant deterioration in performance. However, further reduction in precision down to as low as 3-bit fixed-point results in significant losses in performance. In this paper we propose a new data representation that enables state-of-the-art networks to be encoded to 3 bits with negligible loss in classification performance. To perform this, we take advantage of the fact that the weights and activations in a trained network naturally have non-uniform distributions. Using non-uniform, base-2 logarithmic representation to encode weights, communicate activations, and perform dot-products enables networks to 1) achieve higher classification accuracies than fixed-point at the same resolution and 2) eliminate bulky digital multipliers. Finally, we propose an end-to-end training procedure that uses log representation at 5-bits, which achieves higher final test accuracy than linear at 5-bits.