Learning to forget: Continual prediction with LSTM
Learning to forget: Continual prediction with LSTM
复制标题
DOI：
10.1162/089976600300015015
复制
发表时间：
2000-10-01
影响因子：
2.9
通讯作者：
Cummins, F
中科院分区：
文献类型：
--
作者：
Gers, FA;Schmidhuber, J;Cummins, F
长短期记忆（LSTM; Hochreiter & Schmidhuber，1997）可以解决许多以前的递归神经网络（RNN）学习算法无法解决的任务。我们发现了LSTM网络处理连续输入流的一个弱点，这些输入流没有先验地分割成具有显式标记的端点的连续性，在这些端点处，网络的内部状态可以重置。如果没有重置，状态可能会无限增长，最终导致网络崩溃。我们的补救措施是一种新颖的自适应“遗忘门”，使LSTM单元能够学习在适当的时候重置自己，从而释放内部资源。我们回顾了标准LSTM优于其他RNN算法的示例性基准问题。所有算法（包括LSTM）都无法解决这些问题的连续版本。然而，带有遗忘门的LSTM很容易解决这些问题，而且是以一种优雅的方式。
Long short-term memory (LSTM; Hochreiter & Schmidhuber, 1997) can solve numerous tasks not solvable by previous learning algorithms for recurrent neural networks (RNNs). We identify a weakness of LSTM networks processing continual input streams that are not a priori segmented into subsequences with explicitly marked ends at which the network's internal state could be reset. Without resets, the state may grow indefinitely and eventually cause the network to break down. Our remedy is a novel, adaptive "forget gate" that enables an LSTM cell to learn to reset itself at appropriate times, thus releasing internal resources. We review illustrative benchmark problems on which standard LSTM outperforms other RNN algorithms. All algorithms (including LSTM) fail to solve continual versions of these problems. LSTM with forget gates, however, easily solves them, and in an elegant way.