Memory traces in dynamical systems
Memory traces in dynamical systems
复制标题
DOI：
10.1073/pnas.0804451105
复制
发表时间：
2008-12-02
影响因子：
11.1
通讯作者：
Sompolinsky, Haim
中科院分区：
文献类型：
--
作者：
Ganguli, Surya;Huh, Dongsung;Sompolinsky, Haim
为了对感官输入流进行重要的实时计算，生物系统必须保留近期输入的短期记忆痕迹。一般的高维动态系统在其当前状态下可以保留对过去输入的记忆痕迹。这就提出了一些重要的问题，关于这种记忆轨迹的基本限制以及动力系统达到这些限制所需的性质。我们通过将费雪信息理论应用于由受噪声干扰的时变信号驱动的动态系统来解决这些问题。我们引入Fisher记忆曲线（FMC）作为相对于输入信噪比嵌入在动态状态中的信噪比（SNR）的度量。集成FMC表示内存的总容量。我们将这一理论应用于线性神经网络，并表明具有正常连接矩阵的网络的容量恰好为1，而任何N个神经元的网络的容量最多为N。实现这一界限的非正常网络受到严格的设计约束：它必须具有一个隐藏的前馈结构，该结构将其输入超线性放大N阶，并且输入连接必须最优地匹配该结构。受饱和非线性影响的网络的内存容量进一步受到限制，不能超过根号N。这种限制可以通过具有发散扇形的前馈结构来实现，该结构将信号分布在神经元上，从而避免饱和。我们通过表明流体系统中的记忆可以通过由对流不稳定或湍流开始引起的瞬态非正常放大来维持，从而说明了该理论的普遍性。
To perform nontrivial, real-time computations on a sensory input stream, biological systems must retain a short-term memory trace of their recent inputs. It has been proposed that generic high-dimensional dynamical systems could retain a memory trace for past inputs in their current state. This raises important questions about the fundamental limits of such memory traces and the properties required of dynamical systems to achieve these limits. We address these issues by applying Fisher information theory to dynamical systems driven by time-dependent signals corrupted by noise. We introduce the Fisher Memory Curve (FMC) as a measure of the signal-to-noise ratio (SNR) embedded in the dynamical state relative to the input SNR. The integrated FMC indicates the total memory capacity. We apply this theory to linear neuronal networks and show that the capacity of networks with normal connectivity matrices is exactly 1 and that of any network of N neurons is, at most, N. A nonnormal network achieving this bound is subject to stringent design constraints: It must have a hidden feedforward architecture that superlinearly amplifies its input for a time of order N, and the input connectivity must optimally match this architecture. The memory capacity of networks subject to saturating nonlinearities is further limited, and cannot exceed root N This limit can be realized by feedforward structures with divergent fan out that distributes the signal across neurons, thereby avoiding saturation. We illustrate the generality of the theory by showing that memory in fluid systems can be sustained by transient nonnormal amplification due to convective instability or the onset of turbulence.