Learning compact recurrent neural networks
Learning compact recurrent neural networks
复制标题
DOI：
10.1109/icassp.2016.7472821
复制
发表时间：
2016-03
期刊：
影响因子：
--
通讯作者：
Zhiyun Lu;Vikas Sindhwani;Tara N. Sainath
中科院分区：
文献类型：
--
作者：
Zhiyun Lu;Vikas Sindhwani;Tara N. Sainath
循环神经网络 (RNN)，包括长短期记忆 (LSTM) RNN，已在各种语音识别任务上产生了最先进的结果。然而，这些模型的尺寸通常太大，无法部署在具有内存和延迟限制的移动设备上。在这项工作中，我们研究了通过低秩分解和参数共享方案学习紧凑 RNN 和 LSTM 的机制。我们的目标是研究循环架构中的冗余，在这种架构中可以在不损失性能的情况下允许压缩。人们发现，在底层使用结构化矩阵并在顶层使用共享低秩因子的混合策略特别有效，在 2,000 小时的英语语音搜索任务中，将标准 LSTM 的参数减少了 75%，而 WER 只增加了 0.3%。
Recurrent neural networks (RNNs), including long short-term memory (LSTM) RNNs, have produced state-of-the-art results on a variety of speech recognition tasks. However, these models are often too large in size for deployment on mobile devices with memory and latency constraints. In this work, we study mechanisms for learning compact RNNs and LSTMs via low-rank factorizations and parameter sharing schemes. Our goal is to investigate redundancies in recurrent architectures where compression can be admitted without losing performance. A hybrid strategy of using structured matrices in the bottom layers and shared low-rank factors on the top layers is found to be particularly effective, reducing the parameters of a standard LSTM by 75%, at a small cost of 0.3% increase in WER, on a 2,000-hr English Voice Search task.