Two Routes to Scalable Credit Assignment without Weight Symmetry
Two Routes to Scalable Credit Assignment without Weight Symmetry
复制标题
DOI：
--
复制
发表时间：
2020-02
期刊：
影响因子：
--
通讯作者：
D. Kunin;Aran Nayebi;Javier Sagastuy-Breña;S. Ganguli;Jonathan M. Bloom;Daniel L. K. Yamins
中科院分区：
文献类型：
--
作者：
D. Kunin;Aran Nayebi;Javier Sagastuy-Breña;S. Ganguli;Jonathan M. Bloom;Daniel L. K. Yamins
反向传播的神经可扩展性长期以来一直存在争议，主要是因为它使用了非局部权重传输，即一个神经元瞬时测量另一个神经元突触权重的生物学可疑要求。直到最近，试图创建避免权重传输的本地学习规则通常在反向传播的大规模学习场景中失败，例如使用深度卷积网络的ImageNet分类。在这里，我们研究了最近提出的本地学习规则，产生竞争力的性能与反向传播，并发现它是高度敏感的元paradigm的选择，需要费力的调整，不跨网络架构转移。我们的分析指出了这种不稳定性的根本数学原因，使我们能够确定一个更强大的本地学习规则，更好地转移，而无需元模型调整。尽管如此，我们发现这种局部规则和反向传播之间的性能和稳定性差距随着模型深度的增加而扩大。然后，我们研究了几种非局部学习规则，这些规则将瞬时权重传输的需求放松为更具生物学合理性的“权重估计”过程，表明这些规则与深度网络的最新性能相匹配，并且在存在噪声更新的情况下有效地运行。综上所述，我们的研究结果提出了两条发现无权重对称的信用分配神经实现的途径：进一步改进局部规则，使其在整个体系结构中保持一致，以及识别非局部学习机制的生物实现。
The neural plausibility of backpropagation has long been disputed, primarily for its use of non-local weight transport $-$ the biologically dubious requirement that one neuron instantaneously measure the synaptic weights of another. Until recently, attempts to create local learning rules that avoid weight transport have typically failed in the large-scale learning scenarios where backpropagation shines, e.g. ImageNet categorization with deep convolutional networks. Here, we investigate a recently proposed local learning rule that yields competitive performance with backpropagation and find that it is highly sensitive to metaparameter choices, requiring laborious tuning that does not transfer across network architecture. Our analysis indicates the underlying mathematical reason for this instability, allowing us to identify a more robust local learning rule that better transfers without metaparameter tuning. Nonetheless, we find a performance and stability gap between this local rule and backpropagation that widens with increasing model depth. We then investigate several non-local learning rules that relax the need for instantaneous weight transport into a more biologically-plausible "weight estimation" process, showing that these rules match state-of-the-art performance on deep networks and operate effectively in the presence of noisy updates. Taken together, our results suggest two routes towards the discovery of neural implementations for credit assignment without weight symmetry: further improvement of local rules so that they perform consistently across architectures and the identification of biological implementations for non-local learning mechanisms.