Optimization and applications of echo state networks with leaky-integrator neurons
Optimization and applications of echo state networks with leaky-integrator neurons
复制标题
DOI：
10.1016/j.neunet.2007.04.016
复制
发表时间：
2007-04-01
期刊：
影响因子：
7.8
通讯作者：
Siewert, Udo
中科院分区：
文献类型：
--
作者：
Jaegera, Herbert;Lukosevicius, Mantas;Siewert, Udo
标准回声状态网络 (ESN) 由具有 S 形激活函数的简单加法单元构建。在这里，我们研究了其存储单元是泄漏积分器单元的 ESN。这种类型的单元具有单独的状态动态，可以通过多种方式利用它来使网络适应学习任务的时间特征。我们提出了稳定性条件，介绍并研究了用于优化全局学习参数（输入和输出反馈缩放、泄漏率、谱半径）的随机梯度下降方法，并证明了泄漏积分器 ESN 在以下方面的有用性：（i）学习非常慢的动态系统并以不同的速度重放学习的系统，（ii）对相对缓慢和嘈杂的时间序列进行分类（日语元音数据集 - 在这里我们获得零测试错误率），以及（iii）识别强烈的时间扭曲动态模式。 (c) 2007 Elsevier Ltd. 保留所有权利。
Standard echo state networks (ESNs) are built from simple additive units with a sigmoid activation function. Here we investigate ESNs whose reservoir units are leaky integrator units. Units of this type have individual state dynamics, which can be exploited in various ways to accommodate the network to the temporal characteristics of a learning task. We present stability conditions, introduce and investigate a stochastic gradient descent method for the optimization of the global learning parameters (input and Output feedback scalings, leaking rate, spectral radius) and demonstrate the usefulness of leaky-integrator ESNs for (i) learning very slow dynamic systems and replaying the learnt system at different speeds, (ii) classifying relatively slow and noisy time series (the Japanese Vowel dataset - here we obtain a zero test error rate), and (iii) recognizing strongly time-warped dynamic patterns. (c) 2007 Elsevier Ltd. All rights reserved.