Large Associative Memory Problem in Neurobiology and Machine Learning
Large Associative Memory Problem in Neurobiology and Machine Learning
复制标题
DOI：
--
复制
发表时间：
2020-08
期刊：
影响因子：
--
通讯作者：
D. Krotov;J. Hopfield
中科院分区：
文献类型：
--
作者：
D. Krotov;J. Hopfield
密集关联记忆或现代Hopfield网络允许存储和可靠检索指数级大（在特征空间的维度上）数量的记忆。与此同时，它们的朴素实现是非生物的，因为它似乎需要神经元之间存在多体突触连接。我们表明，这些模型是一个更微观的（书面的生物自由度）理论，有额外的（隐藏的）神经元，只需要两个身体之间的相互作用的有效描述。由于这个原因，我们提出的微观理论是一个有效的模型，大联想记忆与一定程度的生物相容性。我们的网络的动力学和它的降维等效都最小化能量（李雅普诺夫）函数。当某些动态变量（隐藏神经元）从我们的微观理论中整合出来时，可以恢复许多以前在文献中讨论过的模型，例如“Hopfield Networks is All You Need”论文中提出的模型。我们还提供了另一种推导的能量函数和更新规则中提出的上述文件，并澄清这一类的各种模型之间的关系。
Dense Associative Memories or modern Hopfield networks permit storage and reliable retrieval of an exponentially large (in the dimension of feature space) number of memories. At the same time, their naive implementation is non-biological, since it seemingly requires the existence of many-body synaptic junctions between the neurons. We show that these models are effective descriptions of a more microscopic (written in terms of biological degrees of freedom) theory that has additional (hidden) neurons and only requires two-body interactions between them. For this reason our proposed microscopic theory is a valid model of large associative memory with a degree of biological plausibility. The dynamics of our network and its reduced dimensional equivalent both minimize energy (Lyapunov) functions. When certain dynamical variables (hidden neurons) are integrated out from our microscopic theory, one can recover many of the models that were previously discussed in the literature, e.g. the model presented in ''Hopfield Networks is All You Need'' paper. We also provide an alternative derivation of the energy function and the update rule proposed in the aforementioned paper and clarify the relationships between various models of this class.