Compressing recurrent neural network with tensor train
Compressing recurrent neural network with tensor train
复制标题
DOI：
10.1109/ijcnn.2017.7966420
复制
发表时间：
2017-05
期刊：
影响因子：
--
通讯作者：
Andros Tjandra;S. Sakti;Satoshi Nakamura
中科院分区：
文献类型：
--
作者：
Andros Tjandra;S. Sakti;Satoshi Nakamura
循环神经网络 (RNN) 是对时间和顺序任务进行建模的流行选择，并在各种复杂问题上实现了许多最先进的性能。然而，大多数最先进的 RNN 都有数百万个参数，需要大量计算资源来训练和预测新数据。本文提出了一种替代 RNN 模型，通过基于张量训练（TT）格式表示权重参数来显着减少参数数量。在本文中，我们实现了几种 RNN 架构的 TT 格式表示，例如简单 RNN 和门控循环单元 (GRU)。我们在序列分类和序列预测任务上对我们提出的 RNN 模型与未压缩的 RNN 模型进行了比较和评估。我们提出的 TT 格式的 RNN 能够在保持性能的同时将 RNN 参数的数量显着减少 40 倍。
Recurrent Neural Network (RNN) are a popular choice for modeling temporal and sequential tasks and achieve many state-of-the-art performance on various complex problems. However, most of the state-of-the-art RNNs have millions of parameters and require many computational resources for training and predicting new data. This paper proposes an alternative RNN model to reduce the number of parameters significantly by representing the weight parameters based on Tensor Train (TT) format. In this paper, we implement the TT-format representation for several RNN architectures such as simple RNN and Gated Recurrent Unit (GRU). We compare and evaluate our proposed RNN model with uncompressed RNN model on sequence classification and sequence prediction tasks. Our proposed RNNs with TT-format are able to preserve the performance while reducing the number of RNN parameters significantly up to 40 times smaller.