On Sparse Modern Hopfield Model
On Sparse Modern Hopfield Model
复制标题
DOI：
10.48550/arxiv.2309.12673
复制
发表时间：
2023-09
期刊：
影响因子：
--
通讯作者：
Jerry Yao-Chieh Hu;Donglin Yang;Dennis Wu;Chenwei Xu;Bo-Yu Chen;Han Liu
中科院分区：
文献类型：
--
作者：
Jerry Yao-Chieh Hu;Donglin Yang;Dennis Wu;Chenwei Xu;Bo-Yu Chen;Han Liu
作为现代Hopfield模型的稀疏扩展，我们引入了稀疏现代Hopfield模型。与其密集模型一样，稀疏现代Hopfield模型也配备了一个记忆检索动力学，其一步逼近对应于稀疏注意机制。从理论上讲，我们的主要贡献是利用稀疏熵正则化器的凸共轭推导出闭形式稀疏Hopfield能量。在此基础上，我们从稀疏能量函数推导出稀疏记忆检索动力学，并证明其一步逼近等价于稀疏结构注意力。重要的是，我们提供了一个稀疏依赖的内存检索错误边界，可以证明它比密集的模拟更严格。因此，确定并讨论了稀疏性产生的好处的条件。此外，我们还证明了稀疏现代Hopfield模型保持了其密集模型的鲁棒性，包括快速不定点收敛和指数内存容量。经验上，我们使用合成和真实世界的数据集来证明稀疏Hopfield模型在许多情况下优于其密集对应。
We introduce the sparse modern Hopfield model as a sparse extension of the modern Hopfield model. Like its dense counterpart, the sparse modern Hopfield model equips a memory-retrieval dynamics whose one-step approximation corresponds to the sparse attention mechanism. Theoretically, our key contribution is a principled derivation of a closed-form sparse Hopfield energy using the convex conjugate of the sparse entropic regularizer. Building upon this, we derive the sparse memory retrieval dynamics from the sparse energy function and show its one-step approximation is equivalent to the sparse-structured attention. Importantly, we provide a sparsity-dependent memory retrieval error bound which is provably tighter than its dense analog. The conditions for the benefits of sparsity to arise are therefore identified and discussed. In addition, we show that the sparse modern Hopfield model maintains the robust theoretical properties of its dense counterpart, including rapid fixed point convergence and exponential memory capacity. Empirically, we use both synthetic and real-world datasets to demonstrate that the sparse Hopfield model outperforms its dense counterpart in many situations.