Using Fast Weights to Attend to the Recent Past
Using Fast Weights to Attend to the Recent Past
复制标题
DOI：
--
复制
发表时间：
2016-10
期刊：
影响因子：
--
通讯作者：
Jimmy Ba;Geoffrey E. Hinton;Volodymyr Mnih;Joel Z. Leibo;Catalin Ionescu
中科院分区：
文献类型：
--
作者：
Jimmy Ba;Geoffrey E. Hinton;Volodymyr Mnih;Joel Z. Leibo;Catalin Ionescu
直到最近，对人工神经网络的研究在很大程度上仅限于只有两种类型变量的系统：代表当前或最近输入的神经活动和学习捕获输入，输出和收益之间的权重。这种限制没有充分的理由。突触在许多不同的时间尺度上都有动态变化，这表明人工神经网络可能会受益于变化速度比活动慢但比标准权重快得多的变量。这些“快速权重”可以用来存储最近的过去的临时记忆，它们提供了一种神经上可行的方式来实现对过去的注意力，这种注意力最近被证明在序列到序列模型中是有用的。通过使用快速权重，我们可以避免存储神经活动模式的副本。
Until recently, research on artificial neural networks was largely restricted to systems with only two types of variable: Neural activities that represent the current or recent input and weights that learn to capture regularities among inputs, outputs and payoffs. There is no good reason for this restriction. Synapses have dynamics at many different time-scales and this suggests that artificial neural networks might benefit from variables that change slower than activities but much faster than the standard weights. These ``fast weights'' can be used to store temporary memories of the recent past and they provide a neurally plausible way of implementing the type of attention to the past that has recently proven helpful in sequence-to-sequence models. By using fast weights we can avoid the need to store copies of neural activity patterns.