Feature Extraction Mechanism for Each Layer of Deep Echo State Network
Feature Extraction Mechanism for Each Layer of Deep Echo State Network
复制标题
深回波状态网络各层特征提取机制
DOI：
10.1109/icetci55171.2022.9921370
复制
发表时间：
2022
期刊：
影响因子：
--
通讯作者：
Sou Nobukawa
中科院分区：
文献类型：
--
作者：
Keikou Kanda;Sou Nobukawa
回声状态网络（ESN）是一种高效的机器学习模型，是最典型的机器学习计算框架。最近，已经对深度回声状态网络（deepESN）进行了研究。deepESN由一个输入层、多个储层和一个输出层组成，它们实现了非常高的存储容量（MC）。此外，有人建议deepESN可以使用分层来表示各种时间尺度。然而，每一层在特征提取中的确切作用尚未被揭示。因此，必须使用基于试错法的经验测量或网格搜索来进行deepESN参数调整。为了建立deepESN的设计框架，揭示与特征提取相关的deepESN参数至关重要。为了分析神经网络的动力学，我们将多尺度熵（MSE）分析应用于生理神经网络模型，发现复杂的拓扑特征和多个神经模块结构产生复杂的时间尺度依赖性。因此，我们假设可以使用MSE分析来揭示deepESN中每一层的特征提取功能。为了验证这一假设，我们使用MSE分析和MC任务分析了每个层的输出。因此，在这项研究中，在小的内层连接，一个高的记忆容量，实现了时间尺度特定的特征提取在每一层相比，较大的层间连接。总之，本研究揭示了deepESN的特征提取功能，并提供了参数设置方法，特别是将层间连接作为deepESN设计框架的一部分。
The echo state network (ESN) is an efficient machine learning model that is the most typical type of reservoir-computing framework. Recently, research has been conducted on deep echo state networks (deepESN). The deepESN consists of an input layer, multiple reservoir layers, and an output layer, which achieve a very high memory capacity (MC). Furthermore, it has been suggested that deepESN can represent various temporal scales using layer hierarchization. However, the exact role of each layer in the feature extraction has not yet been revealed. Therefore, deepESN parameter adjustments must be conducted using empirical measurements or grid searches based on a trial-and-error method. To establish a design framework for deepESN, revealing the deepESN parameters related to feature extraction is crucial. To analyze the dynamics of neural networks, we applied multiscale entropy (MSE) analysis to a physiological neural network model and found that complex topological features and multiple neural module structures produce complex temporal-scale dependencies. Therefore, we hypothesized that the feature extraction function of each layer in the deepESN could be revealed using MSE analysis. To validate this hypothesis, we analyzed the output of each layer using MSE analysis and MC task. As a result, in this study, under small inner layer connections, a high memory capacity was achieved by temporal scale-specific feature extraction in each layer compared to larger inter-layer connections. In conclusion, this study revealed the feature extraction function in deepESN and provided the parameter setting method, especially regarding interlayer connection as a part of the design framework of deepESN.