Tensor Decomposition for Compressing Recurrent Neural Network
Tensor Decomposition for Compressing Recurrent Neural Network
复制标题
DOI：
10.1109/ijcnn.2018.8489213
复制
发表时间：
2018-02
期刊：
影响因子：
--
通讯作者：
Andros Tjandra;S. Sakti;Satoshi Nakamura
中科院分区：
文献类型：
--
作者：
Andros Tjandra;S. Sakti;Satoshi Nakamura
在机器学习领域，递归神经网络（RNN）已经成为一种流行的序列数据建模架构。然而，在令人印象深刻的性能背后，RNN需要大量的参数来进行训练和推理。在本文中，我们试图减少参数的数量，同时保持RNN的表达能力。我们利用几种张量分解方法，包括CANDECOMP/PARAFAC（CP），Tucker分解和张量训练（TT）重新参数化GRU RNN。我们评估了所有基于张量的RNN在具有各种参数的序列建模任务上的性能。根据我们的实验结果，TT-GRU取得了最好的结果，在各种数量的参数相比，其他分解方法。
In the machine learning fields, Recurrent Neural Network (RNN) has become a popular architecture for sequential data modeling. However, behind the impressive performance, RNNs require a large number of parameters for both training and inference. In this paper, we are trying to reduce the number of parameters and maintain the expressive power from RNN simultaneously. We utilize several tensor decompositions method including CANDECOMP/PARAFAC (CP), Tucker decomposition and Tensor Train (TT) to re-parameterize the Gated Recurrent Unit (GRU) RNN. We evaluate all tensor-based RNNs performance on sequence modeling tasks with a various number of parameters. Based on our experiment results, TT-GRU achieved the best results in a various number of parameters compared to other decomposition methods.