SpaRCe: Improved Learning of Reservoir Computing Systems Through Sparse Representations
SpaRCe: Improved Learning of Reservoir Computing Systems Through Sparse Representations
复制标题
DOI：
10.1109/tnnls.2021.3102378
复制
发表时间：
2021-08-13
影响因子：
10.4
通讯作者：
Vasilaki, Eleni
中科院分区：
文献类型：
--
作者：
Manneschi, Luca;Lin, Andrew C.;Vasilaki, Eleni
“稀疏”神经网络在机器学习和神经科学中都很常见，其中相对较少的神经元或连接是活跃的。虽然在机器学习中，“稀疏性”与导致一些连接权重变小或为零的惩罚项有关，但在生物大脑中，当高尖峰阈值阻止神经元活动时，往往会产生稀疏性。在这里，我们通过神经元特定的可学习活动阈值将稀疏性引入到水库计算网络中，允许低阈值的神经元对决策做出贡献，但抑制来自高阈值神经元的信息。这种方法，我们称之为“SpaRCe”，优化了储集层的稀疏程度，而不影响储集层动态。读出的权重和阈值通过在线梯度规则学习，该在线梯度规则最小化网络输出上的误差函数。阈值学习是通过两种相反的力量的平衡发生的：通过去激活多余的神经元来减少储备库中神经元之间的相关性，同时增加参与正确决策的神经元的活动。我们在分类问题上测试了稀疏算法，发现与标准的油藏计算相比，阈值学习提高了性能。由于增加了网络决策中包含的任务专门化神经元的数量，SPARCE缓解了灾难性遗忘的问题，这一问题在标准回声状态网络(ESNs)和一般的递归神经网络中最为明显。
``Sparse'' neural networks, in which relatively few neurons or connections are active, are common in both machine learning and neuroscience. While, in machine learning, ``sparsity'' is related to a penalty term that leads to some connecting weights becoming small or zero, in biological brains, sparsity is often created when high spiking thresholds prevent neuronal activity. Here, we introduce sparsity into a reservoir computing network via neuron-specific learnable thresholds of activity, allowing neurons with low thresholds to contribute to decision-making but suppressing information from neurons with high thresholds. This approach, which we term ``SpaRCe,'' optimizes the sparsity level of the reservoir without affecting the reservoir dynamics. The read-out weights and the thresholds are learned by an online gradient rule that minimizes an error function on the outputs of the network. Threshold learning occurs by the balance of two opposing forces: reducing interneuronal correlations in the reservoir by deactivating redundant neurons, while increasing the activity of neurons participating in correct decisions. We test SpaRCe on classification problems and find that threshold learning improves performance compared to standard reservoir computing. SpaRCe alleviates the problem of catastrophic forgetting, a problem most evident in standard echo state networks (ESNs) and recurrent neural networks in general, due to increasing the number of task-specialized neurons that are included in the network decisions.