NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks
NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks
复制标题
DOI：
10.48550/arxiv.2206.09117
复制
发表时间：
2022-06
期刊：
影响因子：
--
通讯作者：
Mustafa Burak Gurbuz;C. Dovrolis
中科院分区：
文献类型：
--
作者：
Mustafa Burak Gurbuz;C. Dovrolis
持续学习（CL）的目标是随着时间的推移学习不同的任务。与CL相关的主要期望是保持旧任务的性能，利用后者来改进对未来任务的学习，并在训练过程中引入最小的开销（例如，不需要增长模型或重新训练）。我们提出了神经启发的稳定性-可塑性适应（NISPA）架构，该架构通过固定密度的稀疏神经网络来解决这些需求。NISPA形成稳定的路径来保存从旧任务中学习到的知识。此外，NISPA使用连接重新布线来创建新的塑料路径，从而在新任务中重用现有知识。我们对EMNIST、FashionMNIST、CIFAR10和CIFAR100数据集的广泛评估表明，NISPA显著优于代表性的最先进的持续学习基线，并且与基线相比，它使用的可学习参数减少了10倍。我们还说明了稀疏性是持续学习的基本要素。NISPA代码可在https://github.com/BurakGurbuz97/NISPA上获得。
The goal of continual learning (CL) is to learn different tasks over time. The main desiderata associated with CL are to maintain performance on older tasks, leverage the latter to improve learning of future tasks, and to introduce minimal overhead in the training process (for instance, to not require a growing model or retraining). We propose the Neuro-Inspired Stability-Plasticity Adaptation (NISPA) architecture that addresses these desiderata through a sparse neural network with fixed density. NISPA forms stable paths to preserve learned knowledge from older tasks. Also, NISPA uses connection rewiring to create new plastic paths that reuse existing knowledge on novel tasks. Our extensive evaluation on EMNIST, FashionMNIST, CIFAR10, and CIFAR100 datasets shows that NISPA significantly outperforms representative state-of-the-art continual learning baselines, and it uses up to ten times fewer learnable parameters compared to baselines. We also make the case that sparsity is an essential ingredient for continual learning. The NISPA code is available at https://github.com/BurakGurbuz97/NISPA.