Relating transformers to models and neural representations of the hippocampal formation
Relating transformers to models and neural representations of the hippocampal formation
复制标题
将 Transformer 与海马结构的模型和神经表征联系起来
DOI：
--
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
T. Behrens
中科院分区：
文献类型：
--
作者：
James C. R. Whittington;Joseph Warren;T. Behrens
许多基于大脑网络的深度神经网络架构最近被证明可以复制在大脑中观察到的神经放电模式。变压器神经网络是最令人兴奋和最有前途的新架构之一，它的开发没有考虑到大脑。在这项工作中，我们表明，变压器，当配备周期性的位置编码，复制精确调谐的海马体形成的空间表征；最显著的是位置和网格单元。此外，我们表明这一结果并不令人惊讶，因为它与当前神经科学的海马模型密切相关。我们还展示了变压器版本比神经科学版本提供了戏剧性的性能提升。这项工作继续将人工和大脑网络的计算结合起来，提供了对海马体-皮层相互作用的新理解，并提出了更广泛的皮层区域如何执行比当前神经科学模型（如语言理解）更复杂的任务。
Many deep neural network architectures loosely based on brain networks have recently been shown to replicate neural firing patterns observed in the brain. One of the most exciting and promising novel architectures, the Transformer neural network, was developed without the brain in mind. In this work, we show that transformers, when equipped with recurrent position encodings, replicate the precisely tuned spatial representations of the hippocampal formation; most notably place and grid cells. Furthermore, we show that this result is no surprise since it is closely related to current hippocampal models from neuroscience. We additionally show the transformer version offers dramatic performance gains over the neuroscience version. This work continues to bind computations of artificial and brain networks, offers a novel understanding of the hippocampal-cortical interaction, and suggests how wider cortical areas may perform complex tasks beyond current neuroscience models such as language comprehension.