A biologically inspired architecture with switching units can learn to generalize across backgrounds
A biologically inspired architecture with switching units can learn to generalize across backgrounds
复制标题
DOI：
10.1101/2021.11.08.467807
复制
发表时间：
2021-11
期刊：
影响因子：
--
通讯作者：
Doris Voina;E. Shea-Brown;Stefan Mihalas
中科院分区：
文献类型：
--
作者：
Doris Voina;E. Shea-Brown;Stefan Mihalas
人类和其他动物可以轻松地在不同的景观和环境中导航，这需要大脑能够快速准确地适应不同的视觉领域，并在上下文/背景中进行概括。尽管最近在深度学习方面取得了进展，应用于存在多种混淆（包括上下文混淆）的分类和检测[25，30]，但关于网络如何执行上下文相关计算以及如何形成上下文不变的视觉概念，仍然存在重要的挑战。例如，最近的研究表明，人工网络会反复错误地分类新背景下的熟悉物体，例如，当已知动物出现在不同的环境中时，会错误地标记它们。在这里，我们展示了生物启发的网络基序如何明确地解决这个问题。我们使用一种新的数据集来实现这一点，该数据集可以用作未来研究的基准，以探索背景的不变性。该数据集由不同透明度的MNIST数字组成，设置在具有不同统计数据的两种背景之一上：高斯噪声或来自CIFAR-10数据集的更自然的背景。我们使用这个数据集来学习数字分类，当上下文顺序显示时，发现浅层和深层网络在经验学习第二个背景后返回第一个背景时性能急剧下降-持续学习中的灾难性遗忘现象。为了克服这一点，我们提出了一个架构，在一个新的背景的存在下激活额外的“开关”单元。我们发现，交换网络可以学习新的上下文，即使很少的交换单元，同时保持在以前的上下文中的性能-但它们必须经常连接到网络层。当任务由于高透明度而变得困难时，在两个上下文中训练的切换网络优于仅在一个上下文中训练的没有切换的网络。切换机制导致稀疏的激活模式，我们提供了为什么这有助于解决任务的直觉。我们将我们的架构与其他著名的学习方法进行了比较，发现弹性权重合并在我们的设置中并不成功，而渐进式网络更复杂，但效率较低。因此，我们的研究表明，生物启发的建筑主题如何有助于跨上下文的任务泛化。
Humans and other animals navigate different landscapes and environments with ease, a feat that requires the brain’s ability to rapidly and accurately adapt to different visual domains, generalizing across contexts/backgrounds. Despite recent progress in deep learning applied to classification and detection in the presence of multiple confounds including contextual ones [25, 30], there remain important challenges to address regarding how networks can perform context-dependent computations and how contextually-invariant visual concepts are formed. For instance, recent studies have shown artificial networks that repeatedly misclassified familiar objects set on new backgrounds, e.g. incorrectly labelling known animals when they appeared in a different setting [3]. Here, we show how a bio-inspired network motif can explicitly address this issue. We do this using a novel dataset which can be used as a benchmark for future studies probing invariance to backgrounds. The dataset consists of MNIST digits of varying transparency, set on one of two backgrounds with different statistics: a Gaussian noise or a more naturalistic background from the CIFAR-10 dataset. We use this dataset to learn digit classification when contexts are shown sequentially, and find that both shallow and deep networks have sharply decreased performance when returning to the first background after experience learning the second – the catastrophic forgetting phenomenon in continual learning. To overcome this, we propose an architecture with additional “ switching” units that are activated in the presence of a new background. We find that the switching network can learn the new context even with very few switching units, while maintaining the performance in the previous context – but that they must be recurrently connected to network layers. When the task is difficult due to high transparency, the switching network trained on both contexts outperforms networks without switching trained on only one context. The switching mechanism leads to sparser activation patterns, and we provide intuition for why this helps to solve the task. We compare our architecture with other prominent learning methods, and find that elastic weight consolidation is not successful in our setting, while progressive nets are more complex but less effective. Our study therefore shows how a bio-inspired architectural motif can contribute to task generalization across context.