SIESTA: Efficient Online Continual Learning with Sleep
SIESTA: Efficient Online Continual Learning with Sleep
复制标题
DOI：
10.48550/arxiv.2303.10725
复制
发表时间：
2023-03
期刊：
影响因子：
--
通讯作者：
Md Yousuf Harun;Jhair Gallardo;Tyler L. Hayes;Ronald Kemker;Christopher Kanan
中科院分区：
文献类型：
--
作者：
Md Yousuf Harun;Jhair Gallardo;Tyler L. Hayes;Ronald Kemker;Christopher Kanan
在有监督的持续学习中，深度神经网络（DNN）会随着不断增长的数据流而更新。与数据被打乱的离线设置不同，我们不能对数据流做出任何分布假设。理想情况下，为了提高计算效率，只需要通过数据集一次。然而，现有的方法是不够的，并作出许多假设，不能为现实世界的应用，同时未能提高计算效率。在本文中，我们提出了一种新的持续学习方法，SIESTA基于唤醒/睡眠框架进行训练，这是很好地适应设备上学习的需求。SIESTA的主要目标是提高计算效率的持续学习，以便DNN可以使用更少的时间和能量进行有效更新。SIESTA的主要创新是：1）在唤醒阶段使用无排练，无反向传播和数据驱动的网络更新规则进行快速在线更新，以及2）在睡眠阶段使用计算限制的排练策略加速内存整合。为了提高记忆效率，SIESTA使用来自REMIND的记忆索引来适应潜在复述。与REMIND和现有技术相比，SIESTA的计算效率要高得多，能够在单个GPU上在不到2小时的时间内在ImageNet-1 K上进行持续学习;此外，在无增强设置中，它与离线学习者的性能相匹配，这是推动在现实世界应用中采用持续学习的关键里程碑。
In supervised continual learning, a deep neural network (DNN) is updated with an ever-growing data stream. Unlike the offline setting where data is shuffled, we cannot make any distributional assumptions about the data stream. Ideally, only one pass through the dataset is needed for computational efficiency. However, existing methods are inadequate and make many assumptions that cannot be made for real-world applications, while simultaneously failing to improve computational efficiency. In this paper, we propose a novel continual learning method, SIESTA based on wake/sleep framework for training, which is well aligned to the needs of on-device learning. The major goal of SIESTA is to advance compute efficient continual learning so that DNNs can be updated efficiently using far less time and energy. The principal innovations of SIESTA are: 1) rapid online updates using a rehearsal-free, backpropagation-free, and data-driven network update rule during its wake phase, and 2) expedited memory consolidation using a compute-restricted rehearsal policy during its sleep phase. For memory efficiency, SIESTA adapts latent rehearsal using memory indexing from REMIND. Compared to REMIND and prior arts, SIESTA is far more computationally efficient, enabling continual learning on ImageNet-1K in under 2 hours on a single GPU; moreover, in the augmentation-free setting it matches the performance of the offline learner, a milestone critical to driving adoption of continual learning in real-world applications.