Simplicial Hopfield networks
Simplicial Hopfield networks
复制标题
DOI：
10.48550/arxiv.2305.05179
复制
发表时间：
2023-05
期刊：
影响因子：
--
通讯作者：
Thomas F Burns;T. Fukai
中科院分区：
文献类型：
--
作者：
Thomas F Burns;T. Fukai
Hopfield网络是人工神经网络，其通过选择递归连接权重和更新规则来存储关于其神经元的状态的记忆模式，使得网络的能量景观在记忆周围形成吸引子。在这样一个使用$N$个神经元的网络中，我们可以存储多少稳定的、吸引人的记忆模式？答案取决于权重和更新规则的选择。受生物学中setwise连接的启发，我们通过添加setwise连接并将这些连接嵌入到单纯复形中来扩展Hopfield网络。单纯形复形是图的高维类似物，它自然地表示两两关系和两两关系的集合。我们表明，我们的单纯Hopfield网络增加内存存储容量。令人惊讶的是，即使连接仅限于一个与全成对网络大小相等的小随机子集，我们的网络仍然优于它们的成对网络。这些场景包括非平凡单纯拓扑。我们还测试了类似的现代连续Hopfield网络，为改善Transformer模型中的注意力机制提供了一个潜在的有前途的途径。
Hopfield networks are artificial neural networks which store memory patterns on the states of their neurons by choosing recurrent connection weights and update rules such that the energy landscape of the network forms attractors around the memories. How many stable, sufficiently-attracting memory patterns can we store in such a network using $N$ neurons? The answer depends on the choice of weights and update rule. Inspired by setwise connectivity in biology, we extend Hopfield networks by adding setwise connections and embedding these connections in a simplicial complex. Simplicial complexes are higher dimensional analogues of graphs which naturally represent collections of pairwise and setwise relationships. We show that our simplicial Hopfield networks increase memory storage capacity. Surprisingly, even when connections are limited to a small random subset of equivalent size to an all-pairwise network, our networks still outperform their pairwise counterparts. Such scenarios include non-trivial simplicial topology. We also test analogous modern continuous Hopfield networks, offering a potentially promising avenue for improving the attention mechanism in Transformer models.