The impact of sparsity in low-rank recurrent neural networks.
The impact of sparsity in low-rank recurrent neural networks.
复制标题
DOI：
10.1371/journal.pcbi.1010426
复制
发表时间：
2022-08
影响因子：
4.3
通讯作者：
中科院分区：
文献类型：
--
作者：
神经种群动力学通常是高度协调的，允许与任务相关的计算被理解为通过低维子空间的神经轨迹。网络连通性和输入结构如何导致这种活动可以借助于低阶递归神经网络来研究，低阶递归神经网络是最近发展起来的一类计算模型，它提供了一个丰富的理论框架，将潜在的连通性结构与新兴的低维动力学联系起来。到目前为止，这一框架依赖于所有连接的假设，但众所周知，大脑皮层网络是高度稀疏的。在这里，我们研究连接是随机稀疏的低阶递归网络的动力学，这使得网络的连通性形式上是满阶的。我们首先分析了稀疏性对低阶连通矩阵特征值谱的影响，并以此为基础考察了稀疏性对动力学的影响。我们发现，在稀疏性存在的情况下，复平面上的本征谱由连续的整体和孤立的离群点组成，其形式类似于由低秩随机分量和满秩随机分量组成的连通性矩阵的本征谱。这种类比使我们能够描述稀疏低阶网络作为关键网络参数的函数的不同的动态机制。总之，我们发现，即使在高稀疏性的情况下，由低阶连通结构引起的低维动力学也能被保持，因此即使在稀疏到生物现实的程度的网络中，也可以支持丰富和健壮的计算。在大型神经元网络中，群体表现出的活动取决于每个神经元之间连接的强度。在参与认知任务的大脑皮层区域，这种群体活动通常被视为高度协调和低维的。最近的一系列理论工作探索了这样的协调活动如何出现在神经元网络中，在这个网络中，定义连接的矩阵在数学上被限制为低等级。到目前为止，这种连接结构只在完全连接的网络中被探索过，在这种网络中，每个神经元都相互连接。然而，在大脑中，网络连接通常是高度稀疏的，因为大多数神经元并不共享直接连接。在这里，我们测试了低阶网络理论框架对生物网络中存在的稀疏性现实的稳健性。通过数学分析移除连接的影响，我们发现，以前在密集低阶网络中发现的低维动力学实际上即使在非常高的稀疏性水平上也可以持续存在。这对以下建议有很大的影响：似乎依赖于低维动力学的复杂皮质计算可能得到一个具有根本低等级结构的网络的支持，尽管只有一小部分可能的联系存在。
Neural population dynamics are often highly coordinated, allowing task-related computations to be understood as neural trajectories through low-dimensional subspaces. How the network connectivity and input structure give rise to such activity can be investigated with the aid of low-rank recurrent neural networks, a recently-developed class of computational models which offer a rich theoretical framework linking the underlying connectivity structure to emergent low-dimensional dynamics. This framework has so far relied on the assumption of all-to-all connectivity, yet cortical networks are known to be highly sparse. Here we investigate the dynamics of low-rank recurrent networks in which the connections are randomly sparsified, which makes the network connectivity formally full-rank. We first analyse the impact of sparsity on the eigenvalue spectrum of low-rank connectivity matrices, and use this to examine the implications for the dynamics. We find that in the presence of sparsity, the eigenspectra in the complex plane consist of a continuous bulk and isolated outliers, a form analogous to the eigenspectra of connectivity matrices composed of a low-rank and a full-rank random component. This analogy allows us to characterise distinct dynamical regimes of the sparsified low-rank network as a function of key network parameters. Altogether, we find that the low-dimensional dynamics induced by low-rank connectivity structure are preserved even at high levels of sparsity, and can therefore support rich and robust computations even in networks sparsified to a biologically-realistic extent. In large networks of neurons, the activity displayed by the population depends on the strength of the connections between each neuron. In cortical regions engaged in cognitive tasks, this population activity is often seen to be highly coordinated and low-dimensional. A recent line of theoretical work explores how such coordinated activity can arise in a network of neurons in which the matrix defining the connections is constrained to be mathematically low-rank. Until now, this connectivity structure has only been explored in fully-connected networks, in which every neuron is connected to every other. However, in the brain, network connections are often highly sparse, in the sense that most neurons do not share direct connections. Here, we test the robustness of the theoretical framework of low-rank networks to the reality of sparsity present in biological networks. By mathematically analysing the impact of removing connections, we find that the low-dimensional dynamics previously found in dense low-rank networks can in fact persist even at very high levels of sparsity. This has promising implications for the proposal that complex cortical computations which appear to rely on low-dimensional dynamics may be underpinned by a network which has a fundamentally low-rank structure, albeit with only a small fraction of possible connections present.
登录
查看更多内容
影响因子：
16.2
作者：
Lefort, Sandrine;Tomm, Christian;Petersen, Carl C. H.
通讯作者：
Petersen, Carl C. H.
影响因子：
5.5
作者：
Holmgren, C;Harkany, T;Zilberter, Y
通讯作者：
Zilberter, Y
影响因子：
25
作者：
Ostojic, Srdjan
通讯作者：
Ostojic, Srdjan
影响因子：
16.2
作者：
Pereira, Ulises;Brunel, Nicolas
通讯作者：
Brunel, Nicolas
影响因子：
5.7
作者：
通讯作者：
--