Avoiding catastrophic forgetting by coupling two reverberating neural networks
Avoiding catastrophic forgetting by coupling two reverberating neural networks
复制标题
DOI：
10.1016/s0764-4469(97)82472-9
复制
发表时间：
1997-12-01
期刊：
影响因子：
--
通讯作者：
Rousset, S
中科院分区：
文献类型：
--
作者：
Ans, B;Rousset, S
梯度下降学习过程最常用于神经网络建模。当这些算法（例如反向传播）应用于顺序学习任务时，通常会出现被称为灾难性遗忘（或灾难性干扰）的主要缺点：当已经学习了第一组项目的网络接下来在第二组项目上进行训练时，新学习的信息可能会完全破坏先前学习的信息。为了避免这种令人难以置信的失败，我们提出了一种双网络架构，其中第一个网络同时学习新项目和源自第二个网络的内部伪项目。由于证明伪项目反映了第一个网络先前学习的项目的结构，因此该模型使用旧信息实现了刷新机制。关键的一点是，这种刷新机制是基于回响神经网络，只需要随机刺激即可运行。因此，该模型提供了一种显着减少追溯干扰的方法，同时保留信息本质上的分布式性质，并提出了一种原始但合理的方法，将分布式记忆从大脑的一个位置“复制并粘贴”到另一个位置。
Gradient descent learning procedures are most often used in neural network modeling. When these algorithms (e.g., backpropagation) are applied to sequential learning tasks a major drawback termed catastrophic forgetting (or catastrophic interference), generally arises: when a network having already learned a first set of items is next trained on a second set of items, the newly learned information may completely destroy the information previously learned To avoid this implausible failure, we propose a two-network architecture in which new items are learned by a first network concurrently with internal pseudo-items originating from a second network. As it is demonstrated that the pseudo-items reflect the structure of items previously learned by the first network the model thus implements a refreshing mechanism using the old information. The crucial point is that this refreshing mechanism is based on reverberating neural networks which need only random stimulations to operate. The model thus provides a means to dramatically reduce retroactive interference while conserving the essentially distributed nature of information and proposes an original but plausible means to 'copy and paste' a distributed memory from one place in the brain to another.