Simple Recurrent Units for Highly Parallelizable Recurrence
Simple Recurrent Units for Highly Parallelizable Recurrence
复制标题
DOI：
10.18653/v1/d18-1477
复制
发表时间：
2017-09
期刊：
影响因子：
--
通讯作者：
Tao Lei;Yu Zhang;Sida I. Wang;Huijing Dai;Yoav Artzi
中科院分区：
文献类型：
--
作者：
Tao Lei;Yu Zhang;Sida I. Wang;Huijing Dai;Yoav Artzi
由于并行状态计算的内在困难，常见的循环神经架构的扩展性很差。在这项工作中，我们提出了简单循环单元（SRU），这是一种平衡模型容量和可扩展性的轻循环单元。 SRU 旨在提供富有表现力的递归，实现高度并行化的实现，并进行仔细的初始化以促进深度模型的训练。我们证明了 SRU 在多个 NLP 任务上的有效性。 SRU 在分类和问答数据集上比 cuDNN 优化的 LSTM 实现了 5-9 倍的加速，并提供了比 LSTM 和卷积模型更强的结果。通过将 SRU 纳入架构，我们在翻译方面比 Transformer 模型（Vaswani 等人，2017）平均获得了 0.7 BLEU 的改进。
Common recurrent neural architectures scale poorly due to the intrinsic difficulty in parallelizing their state computations. In this work, we propose the Simple Recurrent Unit (SRU), a light recurrent unit that balances model capacity and scalability. SRU is designed to provide expressive recurrence, enable highly parallelized implementation, and comes with careful initialization to facilitate training of deep models. We demonstrate the effectiveness of SRU on multiple NLP tasks. SRU achieves 5—9x speed-up over cuDNN-optimized LSTM on classification and question answering datasets, and delivers stronger results than LSTM and convolutional models. We also obtain an average of 0.7 BLEU improvement over the Transformer model (Vaswani et al., 2017) on translation by incorporating SRU into the architecture.