Recurrent Neural Networks With Limited Numerical Precision
Recurrent Neural Networks With Limited Numerical Precision
复制标题
DOI：
--
复制
发表时间：
2016-08
期刊：
影响因子：
--
通讯作者：
Joachim Ott;Zhouhan Lin;Y. Zhang;Shih-Chii Liu;Yoshua Bengio
中科院分区：
文献类型：
--
作者：
Joachim Ott;Zhouhan Lin;Y. Zhang;Shih-Chii Liu;Yoshua Bengio
递归神经网络（RNN）在许多机器学习任务上具有最先进的性能，但它们对内存和计算能力方面的资源需求通常很高。因此，人们对优化使用这些模型执行的计算非常感兴趣，特别是在考虑为深度网络开发专用低功耗硬件时。减少计算需求的一种方法是限制网络权重和偏差的数值精度。这导致了不同的舍入方法，到目前为止，这些方法仅应用于卷积神经网络和全连接网络。本文讨论了在RNN的情况下如何在训练过程中最好地降低权重精度的问题。我们提出了使用不同的随机和确定性降低精度训练方法的结果，这些方法应用于三种主要的RNN类型，然后在几个数据集上进行测试。结果表明，加权二值化方法不适用于RNN。然而，随机和确定性三端化以及幂端化方法产生了低精度的RNN，这些RNN在某些数据集上产生了类似甚至更高的精度，因此为在专用硬件中训练更有效的RNN实现提供了一条途径。
Recurrent Neural Networks (RNNs) produce state-of-art performance on many machine learning tasks but their demand on resources in terms of memory and computational power are often high. Therefore, there is a great interest in optimizing the computations performed with these models especially when considering development of specialized low-power hardware for deep networks. One way of reducing the computational needs is to limit the numerical precision of the network weights and biases. This has led to different proposed rounding methods which have been applied so far to only Convolutional Neural Networks and Fully-Connected Networks. This paper addresses the question of how to best reduce weight precision during training in the case of RNNs. We present results from the use of different stochastic and deterministic reduced precision training methods applied to three major RNN types which are then tested on several datasets. The results show that the weight binarization methods do not work with the RNNs. However, the stochastic and deterministic ternarization, and pow2-ternarization methods gave rise to low-precision RNNs that produce similar and even higher accuracy on certain datasets therefore providing a path towards training more efficient implementations of RNNs in specialized hardware.