Preventing Deterioration of Classification Accuracy in Predictive Coding Networks
Preventing Deterioration of Classification Accuracy in Predictive Coding Networks
复制标题
DOI：
10.48550/arxiv.2208.07114
复制
发表时间：
2022-08
期刊：
影响因子：
--
通讯作者：
Paul F Kinghorn;Beren Millidge;C. Buckley
中科院分区：
文献类型：
--
作者：
Paul F Kinghorn;Beren Millidge;C. Buckley
预测编码网络（PCNs）旨在学习世界的生成模型。给定观察结果，这个生成模型可以反过来推断这些观察结果的原因。然而，当训练pcn时，通常会观察到一个明显的病理现象，即推理准确率达到峰值，然后随着进一步训练而下降。这不能用过拟合来解释，因为训练和测试精度同时下降。在这里，我们对这一现象进行了彻底的研究，并表明它是由PCN各层收敛速度之间的不平衡引起的。我们证明，这可以通过正则化每层的权重矩阵来防止：通过限制矩阵奇异值的相对大小，我们允许权重矩阵改变，但限制了一层对其相邻层的总体影响。我们也证明了类似的效果可以通过一个生物学上更合理和简单的方案来实现。
Predictive Coding Networks (PCNs) aim to learn a generative model of the world. Given observations, this generative model can then be inverted to infer the causes of those observations. However, when training PCNs, a noticeable pathology is often observed where inference accuracy peaks and then declines with further training. This cannot be explained by overfitting since both training and test accuracy decrease simultaneously. Here we provide a thorough investigation of this phenomenon and show that it is caused by an imbalance between the speeds at which the various layers of the PCN converge. We demonstrate that this can be prevented by regularising the weight matrices at each layer: by restricting the relative size of matrix singular values, we allow the weight matrix to change but restrict the overall impact which a layer can have on its neighbours. We also demonstrate that a similar effect can be achieved through a more biologically plausible and simple scheme of just capping the weights.