Stabilizing patterns in time: Neural network approach.
Stabilizing patterns in time: Neural network approach.
复制标题
DOI：
10.1371/journal.pcbi.1005861
复制
发表时间：
2017-12
影响因子：
4.3
通讯作者：
Tsodyks M
中科院分区：
文献类型：
--
作者：
Ben-Shushan N;Tsodyks M
循环和反馈网络能够保持动态记忆。尽管如此，为这项任务培训一个网络是具有挑战性的。为了做到这一点，人们应该面对系统中误差的非线性传播。由于误差或固有噪声导致的与所需动态的微小偏差可能会在未来产生显著影响。因此，需要一种方法来应对这些困难。在这项工作中，我们主要研究具有线性激活函数和二进制输出单元的递归网络。我们描述了它在其输出单元上再现动作的时间序列的能力。我们建议将时间学习问题转化为感知器问题。在离散情况下，出现有限裕度，在某种程度上为网络提供了对噪声的稳健性，对于这一点，它执行得很好(即，为任意数量的周期完美地产生所需的序列)。在连续情况下，当输出单元改变其状态时，裕度接近于零，因此网络仅能够以轻微抖动来再现序列。数值模拟表明，在离散时间情况下，可学习的最长序列的规模充其量是网络大小的平方根。当并行学习几个短序列时，即它们的总长度大大超过网络可以学习的最长单个序列的长度时，就会产生戏剧性的效果。这种模型很容易推广到任意数量的输出单元，从而提高了它的性能。通过两个序列学习的实际例子说明了这一效果。这项工作为训练递归网络提供了一种克服稳定性问题的方法，并进一步量化了在特定学习方案下网络的性能。在精细的时间分辨率下学习和执行动作的能力至关重要，因为我们的许多日常动作都需要这样的时间顺序(例如肢体运动和语音)。事实上，使用神经网络生成稳定的时变输出在过去几年中吸引了很多关注。当面对这样的任务时，核心问题之一是解的稳定性，因此只能在有限的循环次数内产生序列。在这里，我们提出了一种学习时变序列的稳健方法。
Recurrent and feedback networks are capable of holding dynamic memories. Nonetheless, training a network for that task is challenging. In order to do so, one should face non-linear propagation of errors in the system. Small deviations from the desired dynamics due to error or inherent noise might have a dramatic effect in the future. A method to cope with these difficulties is thus needed. In this work we focus on recurrent networks with linear activation functions and binary output unit. We characterize its ability to reproduce a temporal sequence of actions over its output unit. We suggest casting the temporal learning problem to a perceptron problem. In the discrete case a finite margin appears, providing the network, to some extent, robustness to noise, for which it performs perfectly (i.e. producing a desired sequence for an arbitrary number of cycles flawlessly). In the continuous case the margin approaches zero when the output unit changes its state, hence the network is only able to reproduce the sequence with slight jitters. Numerical simulation suggest that in the discrete time case, the longest sequence that can be learned scales, at best, as square root of the network size. A dramatic effect occurs when learning several short sequences in parallel, that is, their total length substantially exceeds the length of the longest single sequence the network can learn. This model easily generalizes to an arbitrary number of output units, which boost its performance. This effect is demonstrated by considering two practical examples for sequence learning. This work suggests a way to overcome stability problems for training recurrent networks and further quantifies the performance of a network under the specific learning scheme. The ability to learn and execute actions in fine temporal resolution is crucial, as many of our day to day actions require such temporal ordering (e.g. limb movement and speech). Indeed, generating stable time-varying outputs, using neural networks has attracted a lot of attention over the last years. One of the core problems, when facing such a task, is the solution stability, hence it was only possible to produce the sequence for a limited number of cycles. Here we propose a robust approach for the task of learning time-varying sequences.
登录
查看更多内容
DOI：
10.1088/0305-4470/25/16/014
发表时间：
1992-08-21
期刊：
JOURNAL OF PHYSICS A-MATHEMATICAL AND GENERAL
影响因子：
--
作者：
BRESSLOFF, PC;TAYLOR, JG
通讯作者：
TAYLOR, JG
影响因子：
16.2
作者：
Sussillo, David;Abbott, L. F.
通讯作者：
Abbott, L. F.
影响因子：
7.5
作者：
CORTES, C;VAPNIK, V
通讯作者：
VAPNIK, V
影响因子：
2.9
作者：
Adini, Yael;Bonneh, Yoram S.;Israeli, David
通讯作者：
Israeli, David
影响因子：
6
作者：
Schiller, UD;Steil, JJ
通讯作者：
Steil, JJ