A pruning based method to learn both weights and connections for LSTM
A pruning based method to learn both weights and connections for LSTM
复制标题
一种基于剪枝的方法来学习 LSTM 的权重和连接
DOI：
--
复制
发表时间：
2015
期刊：
影响因子：
--
通讯作者：
Jianglei Han
中科院分区：
文献类型：
--
作者：
Shijian Tang;Jianglei Han
这个项目是威廉·达利教授团队的研究课题之一。在这个项目中，我们开发了一种基于剪枝的方法来学习长期短期记忆(LSTM)的权重和连接。在该方法中，我们去掉了预先训练的LSTM中不重要的连接，并使权重矩阵稀疏化。然后，我们对剩余的模型进行重新训练。在剩余的模型收敛后，我们再次剪枝该模型，并迭代地重新训练剩余的模型，直到我们达到期望的模型大小和性能。这种方法将节省LSTM的大小，并防止过度匹配。我们在NeuralTalk上的再训练结果表明，我们可以丢弃近90%的权重，而不会对性能造成太大影响。该项目的部分成果将于2015年公布。
This project is one of the research topics in Professor William Dally’s group. In this project, we developed a pruning based method to learn both weights and connections for Long Short Term Memory (LSTM). In this method, we discard the unimportant connections in a pretrained LSTM, and make the weight matrix sparse. Then, we retrain the remaining model. After we remaining model is converge, we prune this model again and retrain the remaining model iteratively, until we achieve the desired size of model and performance. This method will save the size of the LSTM as well as prevent overfitting. Our results retrained on NeuralTalk shows that we can discard nearly 90% of the weights without hurting the performance too much. Part of the results in this project will be posted in NIPS 2015.