Schematic Memory Persistence and Transience for Efficient and Robust Continual Learning
Schematic Memory Persistence and Transience for Efficient and Robust Continual Learning
复制标题
DOI：
10.1016/j.neunet.2021.08.011
复制
发表时间：
2021-05
期刊：
影响因子：
--
通讯作者：
Yuyang Gao;G. Ascoli;Liang Zhao
中科院分区：
文献类型：
--
作者：
Yuyang Gao;G. Ascoli;Liang Zhao
持续学习被认为是迈向下一代人工智能（AI）的有希望的一步，其中深度神经网络（DNN）通过持续学习类似于人类学习过程的一系列不同任务来做出决策。它仍然相当原始，现有的工作主要集中在避免（灾难性的）遗忘。然而，由于遗忘是不可避免的，给定有限的记忆和无限的任务负载，“如何合理地遗忘”是一个问题，持续学习必须解决，以减少人工智能和人类之间的性能差距，在（1）记忆效率，（2）概括性，和（3）处理噪声数据时的鲁棒性。为了解决这个问题，我们提出了一种新的ScheMATic记忆持久性和短暂性（SMART）1框架，用于基于神经科学最新进展的外部记忆的持续学习。一种新的长期遗忘机制和图式记忆，使用稀疏性和“向后正迁移”的限制与理论保证的错误界的效率和泛化能力提高。鲁棒增强是通过使用一种新的短期遗忘机制，灵感来自背景信息门控学习。最后，在基准数据集和真实数据集上进行了大量的实验分析，证明了该模型的有效性和效率。
Continual learning is considered a promising step toward next-generation Artificial Intelligence (AI), where deep neural networks (DNNs) make decisions by continuously learning a sequence of different tasks akin to human learning processes. It is still quite primitive, with existing works focusing primarily on avoiding (catastrophic) forgetting. However, since forgetting is inevitable given bounded memory and unbounded task loads, ‘how to reasonably forget’ is a problem continual learning must address in order to reduce the performance gap between AIs and humans, in terms of (1) memory efficiency, (2) generalizability, and (3) robustness when dealing with noisy data. To address this, we propose a novel ScheMAtic memory peRsistence and Transience (SMART)1framework for continual learning with external memory that builds on recent advances in neuroscience. The efficiency and generalizability are enhanced by a novel long-term forgetting mechanism and schematic memory, using sparsity and ‘backward positive transfer’ constraints with theoretical guarantees on the error bound. Robust enhancement is achieved using a novel short-term forgetting mechanism inspired by background information-gated learning. Finally, an extensive experimental analysis on both benchmark and real-world datasets demonstrates the effectiveness and efficiency of our model.