MomentumRNN: Integrating Momentum into Recurrent Neural Networks
MomentumRNN: Integrating Momentum into Recurrent Neural Networks
复制标题
DOI：
--
复制
发表时间：
2020-06
期刊：
影响因子：
--
通讯作者：
T. Nguyen;Richard Baraniuk;A. Bertozzi;S. Osher;Baorui Wang
中科院分区：
文献类型：
--
作者：
T. Nguyen;Richard Baraniuk;A. Bertozzi;S. Osher;Baorui Wang
设计深神经网络是一门艺术，通常涉及对候选体系结构进行昂贵的搜索。为了克服复发性神经网（RNN），我们建立了RNN中隐藏状态动力学和梯度下降（GD）之间的联系。然后，我们将动量整合到该框架中，并提出了一个新的RNN家族，称为{\ em Momentumrnns}。从理论上讲，我们证明并在数值上证明了势头减轻培训RNNS中消失的梯度问题。我们研究了动量长期记忆（动量），并验证其在跨LSTM对应方面的收敛速度和准确性方面的优势，在各种基准测试中，计算或记忆效率几乎没有妥协。我们还证明了动量适用于许多类型的复发细胞，包括最新的正交RNN中的细胞。最后，我们表明，可以轻松地将基于先进的基于动量的优化方法（例如ADAM和NESTEROV加速梯度加速梯度）可以轻松地纳入动量框架中，以设计具有更好性能的新的复发单元。该代码可在\ url {this HTTPS url}上获得。
Designing deep neural networks is an art that often involves an expensive search over candidate architectures. To overcome this for recurrent neural nets (RNNs), we establish a connection between the hidden state dynamics in an RNN and gradient descent (GD). We then integrate momentum into this framework and propose a new family of RNNs, called {\em MomentumRNNs}. We theoretically prove and numerically demonstrate that MomentumRNNs alleviate the vanishing gradient issue in training RNNs. We study the momentum long-short term memory (MomentumLSTM) and verify its advantages in convergence speed and accuracy over its LSTM counterpart across a variety of benchmarks, with little compromise in computational or memory efficiency. We also demonstrate that MomentumRNN is applicable to many types of recurrent cells, including those in the state-of-the-art orthogonal RNNs. Finally, we show that other advanced momentum-based optimization methods, such as Adam and Nesterov accelerated gradients with a restart, can be easily incorporated into the MomentumRNN framework for designing new recurrent cells with even better performance. The code is available at \url{this https URL}.