Incremental Task Learning with Incremental Rank Updates-猫眼课题宝

DOI：

10.48550/arxiv.2207.09074

发表时间：

2022-07

期刊：

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Rakib Hyder;Ken Shao;Boyu Hou;P. Markopoulos;Ashley Prater-Bennette;M. Salman Asif

增量任务学习（ITL）是一种持续学习，旨在为多个任务（一个接一个）训练单个网络，其中每个任务的训练数据仅在该任务的训练期间可用。神经网络在接受新任务训练时往往会忘记旧任务;这种特性通常被称为灾难性遗忘。为解决这一问题，国际交易日志方法使用情景记忆、参数正规化、掩蔽和修剪或可扩展网络结构。本文提出了一种新的基于低秩因子分解的增量任务学习框架。特别地，我们将每层的网络权重表示为几个秩为1的矩阵的线性组合。为了更新网络以执行新任务，我们学习一个秩为1（或低秩）的矩阵，并将其添加到每一层的权重中。我们还引入了一个额外的选择器向量，它为之前任务学习的低秩矩阵分配不同的权重。我们表明，我们的方法在准确性和遗忘方面比目前最先进的方法表现得更好。我们的方法还提供了更好的记忆效率相比，情景记忆和面具为基础的方法。我们的代码将在https://github.com/CSIPlab/task-increment-rank-update.git上提供

Incremental Task learning (ITL) is a category of continual learning that seeks to train a single network for multiple tasks (one after another), where training data for each task is only available during the training of that task. Neural networks tend to forget older tasks when they are trained for the newer tasks; this property is often known as catastrophic forgetting. To address this issue, ITL methods use episodic memory, parameter regularization, masking and pruning, or extensible network structures. In this paper, we propose a new incremental task learning framework based on low-rank factorization. In particular, we represent the network weights for each layer as a linear combination of several rank-1 matrices. To update the network for a new task, we learn a rank-1 (or low-rank) matrix and add that to the weights of every layer. We also introduce an additional selector vector that assigns different weights to the low-rank matrices learned for the previous tasks. We show that our approach performs better than the current state-of-the-art methods in terms of accuracy and forgetting. Our method also offers better memory efficiency compared to episodic memory- and mask-based approaches. Our code will be available at https://github.com/CSIPlab/task-increment-rank-update.git

Incremental Task Learning with Incremental Rank Updates

文献摘要

被引文献

相似文献