A Continual Learning Survey: Defying Forgetting in Classification Tasks
A Continual Learning Survey: Defying Forgetting in Classification Tasks
复制标题
DOI：
10.1109/tpami.2021.3057446
复制
发表时间：
2022-07-01
影响因子：
23.6
通讯作者：
Tuytelaars, Tinne
中科院分区：
文献类型：
--
作者：
De Lange, Matthias;Aljundi, Rahaf;Tuytelaars, Tinne
人工神经网络在解决特定刚性任务的分类问题方面非常活跃，它通过从不同的训练阶段概括学习行为来获取知识。由此产生的网络类似于一个静态的知识实体，努力在不针对原始任务的情况下扩展这些知识，从而导致灾难性的遗忘。持续的学习将这一范式转变为可以在不同任务中持续积累知识的网络，而不需要从头开始重新培训。我们专注于任务增量分类，其中任务按顺序到达，并由清晰的边界来描述。我们的主要贡献涉及：(1)分类和对最新技术的广泛概述；(2)持续确定持续学习者稳定性-可塑性权衡的新框架；(3)11种最先进的持续学习方法的综合实验比较；以及(4)基线。考虑到微小的Imagenet和大规模的不平衡iNaturist和一系列识别数据集，我们在三个基准测试上经验地仔细审查了方法的优点和缺点。我们研究了模型容量、权重衰减和丢弃正则化以及任务的呈现顺序对算法的影响，并从所需内存、计算时间和存储空间等方面对这些方法进行了定性比较。
Artificial neural networks thrive in solving the classification problem for a particular rigid task, acquiring knowledge through generalized learning behaviour from a distinct training phase. The resulting network resembles a static entity of knowledge, with endeavours to extend this knowledge without targeting the original task resulting in a catastrophic forgetting. Continual learning shifts this paradigm towards networks that can continually accumulate knowledge over different tasks without the need to retrain from scratch. We focus on task incremental classification, where tasks arrive sequentially and are delineated by clear boundaries. Our main contributions concern: (1) a taxonomy and extensive overview of the state-of-the-art; (2) a novel framework to continually determine the stability-plasticity trade-off of the continual learner; (3) a comprehensive experimental comparison of 11 state-of-the-art continual learning methods; and (4) baselines. We empirically scrutinize method strengths and weaknesses on three benchmarks, considering Tiny Imagenet and large-scale unbalanced iNaturalist and a sequence of recognition datasets. We study the influence of model capacity, weight decay and dropout regularization, and the order in which the tasks are presented, and qualitatively compare methods in terms of required memory, computation time, and storage.