Learning with Partial Forgetting in Modern Hopfield Networks
Learning with Partial Forgetting in Modern Hopfield Networks
复制标题
DOI：
--
复制
发表时间：
2023
期刊：
影响因子：
--
通讯作者：
Toshihiro Ota;Ikuro Sato;Rei Kawakami;Masayuki Tanaka;Nakamasa Inoue
中科院分区：
文献类型：
--
作者：
Toshihiro Ota;Ikuro Sato;Rei Kawakami;Masayuki Tanaka;Nakamasa Inoue
神经科学研究已经知道，部分和短暂的遗忘记忆通常在大脑中起着重要的作用，以改善某些智力活动的性能。作为封闭的重复网络的特征空间中的吸引者。功能是由元素的非主体项目设计的，用于现代跳跃网络中的内存神经元，以改善模型性能。网络，通过修改相应的Lagrangian。用于计算生物学的类别和计算机视觉的图像分类，并确认LWPF始终提高现有神经网络的性能，包括DeepRC和Vision Transformers。
It has been known by neuroscience studies that partial and transient forgetting of memory often plays an important role in the brain to improve performance for certain intellectual activities. In machine learning, associative memory models such as classical and modern Hopfield networks have been proposed to express memories as at-tractors in the feature space of a closed recurrent network. In this work, we propose learning with partial forgetting (LwPF), where a partial forgetting functionality is designed by element-wise non-bijective projections, for memory neurons in modern Hopfield networks to improve model performance. We incorporate LwPF into the attention mechanism also, whose process has been shown to be identical to the update rule of a certain modern Hopfield network, by modifying the corresponding Lagrangian. We evaluated the effectiveness of LwPF on three diverse tasks such as bit-pattern classification, immune repertoire clas-sification for computational biology, and image classification for computer vision, and confirmed that LwPF consistently improves the performance of existing neural networks including DeepRC and vision transformers.