How connectivity structure shapes rich and lazy learning in neural circuits
How connectivity structure shapes rich and lazy learning in neural circuits
复制标题
DOI：
10.48550/arxiv.2310.08513
复制
发表时间：
2023-10
期刊：
影响因子：
--
通讯作者：
Yuhan Helena Liu;A. Baratin;Jonathan Cornford;Stefan Mihalas;E. Shea-Brown;Guillaume Lajoie
中科院分区：
文献类型：
--
作者：
Yuhan Helena Liu;A. Baratin;Jonathan Cornford;Stefan Mihalas;E. Shea-Brown;Guillaume Lajoie
在理论神经科学中，最近的工作利用深度学习工具来探索一些网络属性如何严重影响其学习动态。值得注意的是，初始权重分布较小（分别为大）方差可以产生丰富的（或懒惰）制度，其中显着（分别）在学习过程中观察到网络状态和表示的微小）变化。然而，在生物学中，神经回路连接可能表现出低秩结构，因此与通常用于这些研究的随机初始化明显不同。因此，在这里我们研究初始权重的结构-特别是它们的有效秩-如何影响网络学习机制。通过实证和理论分析，我们发现高阶初始化通常会产生较小的网络变化，这表明学习更懒惰，这一发现也证实了递归神经网络中实验驱动的初始连接。相反，低秩初始化使学习偏向于更丰富的学习。然而，重要的是，作为这一规则的例外，我们发现懒惰的学习仍然可以发生在与任务和数据统计相一致的低秩初始化中。我们的研究强调了初始体重结构在塑造学习机制中的关键作用，以及对可塑性代谢成本和灾难性遗忘风险的影响。
In theoretical neuroscience, recent work leverages deep learning tools to explore how some network attributes critically influence its learning dynamics. Notably, initial weight distributions with small (resp. large) variance may yield a rich (resp. lazy) regime, where significant (resp. minor) changes to network states and representation are observed over the course of learning. However, in biology, neural circuit connectivity could exhibit a low-rank structure and therefore differs markedly from the random initializations generally used for these studies. As such, here we investigate how the structure of the initial weights -- in particular their effective rank -- influences the network learning regime. Through both empirical and theoretical analyses, we discover that high-rank initializations typically yield smaller network changes indicative of lazier learning, a finding we also confirm with experimentally-driven initial connectivity in recurrent neural networks. Conversely, low-rank initialization biases learning towards richer learning. Importantly, however, as an exception to this rule, we find lazier learning can still occur with a low-rank initialization that aligns with task and data statistics. Our research highlights the pivotal role of initial weight structures in shaping learning regimes, with implications for metabolic costs of plasticity and risks of catastrophic forgetting.