Path-Normalized Optimization of Recurrent Neural Networks with ReLU Activations
Path-Normalized Optimization of Recurrent Neural Networks with ReLU Activations
复制标题
DOI：
--
复制
发表时间：
2016-05
期刊：
影响因子：
--
通讯作者：
Behnam Neyshabur;Yuhuai Wu;R. Salakhutdinov;N. Srebro
中科院分区：
文献类型：
--
作者：
Behnam Neyshabur;Yuhuai Wu;R. Salakhutdinov;N. Srebro
我们研究了递归神经网络（RNN）的参数空间几何结构，并开发了一种适应这种几何结构的路径SGD优化方法，该方法可以学习具有ReLU激活的普通RNN。在几个需要捕获长期依赖结构的数据集上，我们表明，与使用SGD训练的RNN相比，即使使用各种最近建议的初始化方案，path-SGD也可以显着提高ReLU RNN的可训练性。
We investigate the parameter-space geometry of recurrent neural networks (RNNs), and develop an adaptation of path-SGD optimization method, attuned to this geometry, that can learn plain RNNs with ReLU activations. On several datasets that require capturing long-term dependency structure, we show that path-SGD can significantly improve trainability of ReLU RNNs compared to RNNs trained with SGD, even with various recently suggested initialization schemes.