A biologically inspired architecture with switching units can learn to generalize across backgrounds.
A biologically inspired architecture with switching units can learn to generalize across backgrounds.
复制标题
具有切换单元的受生物学启发的架构可以学习跨背景的概括。
DOI：
10.1016/j.neunet.2023.09.014
复制
发表时间：
2023
期刊：
影响因子：
--
通讯作者：
Mihalas,Stefan
中科院分区：
文献类型：
--
作者：
Voina,Doris;Shea-Brown,Eric;Mihalas,Stefan
人类和其他动物毫不费力地在不同的环境中导航，他们的大脑迅速而准确地概括了不同的环境。尽管最近在深度学习方面取得了进展，但这种灵活性仍然是许多人工系统面临的挑战。在这里，我们展示了生物启发的网络基序如何明确地解决这个问题。我们使用不同透明度的MNIST数字数据集来实现这一点，该数据集设置在定义两个上下文的不同统计数据的两个背景之一上：像素噪声或来自CIFAR-10数据集的更自然的背景。在学习数字分类后，当两个上下文依次显示时，我们发现浅层和深层网络在返回到第一个背景时性能都急剧下降-这是持续学习中已知的灾难性遗忘现象的一个实例。为了克服这一点，我们提出了一种校验交换网络或简称交换网络。这是一个生物启发的架构，类似于视觉皮层中的一个经过充分研究的网络基序，具有额外的“开关”单元，这些单元在新背景的存在下被激活，假设先验的上下文信号来打开或关闭这些单元。有趣的是，这些切换单元中只有少数几个足以使网络通过抑制冗余背景特征来学习新的上下文，而不会发生灾难性的遗忘。此外，该网络还可以推广到与它所学习的上下文类似的新上下文。重要的是，我们发现，-再次在底层的生物网络基序，recurrentlyconnecting开关单元的网络层是有利的上下文泛化。
Humans and other animals navigate different environments effortlessly, their brains rapidly and accurately generalizing across contexts. Despite recent progress in deep learning, this flexibility remains a challenge for many artificial systems. Here, we show how a bio-inspired network motif can explicitly address this issue. We do this using a dataset of MNIST digits of varying transparency, set on one of two backgrounds of different statistics that define two contexts: a pixel-wise noise or a more naturalistic background from the CIFAR-10 dataset. After learning digit classification when both contexts are shown sequentially, we find that both shallow and deep networks have sharply decreased performance when returning to the first background — an instance of the catastrophic forgetting phenomenon known from continual learning. To overcome this, we propose the bottleneck-switching network or switching network for short. This is a bio-inspired architecture analogous to a well-studied network motif in the visual cortex, with additional “switching” units that are activated in the presence of a new background, assuming a priori a contextual signal to turn these units on or off. Intriguingly, only a few of these switching units are sufficient to enable the network to learn the new context without catastrophic forgetting through inhibition of redundant background features. Further, the bottleneck-switching network can generalize to novel contexts similar to contexts it has learned. Importantly, we find that — again as in the underlying biological network motif,recurrentlyconnecting the switching units to network layers is advantageous for context generalization.