Deep Incremental RNN for Learning Sequential Data: A Lyapunov Stable Dynamical System
Deep Incremental RNN for Learning Sequential Data: A Lyapunov Stable Dynamical System
复制标题
DOI：
10.1109/icdm51629.2021.00108
复制
发表时间：
2021-12
期刊：
影响因子：
--
通讯作者：
Ziming Zhang;Guojun Wu;Yanhua Li;Yun Yue;Xun Zhou
中科院分区：
文献类型：
--
作者：
Ziming Zhang;Guojun Wu;Yanhua Li;Yun Yue;Xun Zhou
随着移动的传感技术的最新进展，收集了大量的连续数据，例如车辆GPS记录、股票价格、来自空气质量检测器的传感器数据。循环神经网络（RNN）已被广泛研究，以学习序列数据的复杂模式，并应用于句子预测/完成的自然语言处理、预测或分类人类活动的人类活动识别。然而，在训练RNN时存在许多实际问题，例如，由于网络权重的重复性等原因，经常会出现梯度消失和爆炸现象。本文研究了深度递归神经网络（RNN）的训练稳定性，并提出了一种新的网络，即深度增量RNN（DIRNN）。与文献相比，我们证明了DIRNN本质上是一个李雅普诺夫稳定的动力系统，在训练中没有消失或爆炸梯度。为了证明在实践中的适用性，我们还提出了一种新的实现，即TinyRNN，它使用加权随机排列来稀疏DIRNN中的转移矩阵，以减少模型大小。我们在七个基准数据集上评估了我们的方法，并取得了最先进的结果。演示代码在补充文件中提供。
With the recent advances in mobile sensing technologies, large amounts of sequential data are collected, such as vehicle GPS records, stock prices, sensor data from air quality detectors. Recurrent neural networks (RNNs) have been studied extensively to learn complex patterns for sequential data, with applicatons in natural language processing for sentence prediction/completion, human activity recognition for predicting or classifying human activities. However, there are many practical issues when training RNNs, e.g., vanishing and exploding gradients often occur due to the repeatability of network weights, etc. In this paper, we study the training stability in deep recurrent neural networks (RNNs), and propose a novel network, namely, deep incremental RNN (DIRNN). In contrast to the literature, we prove that DIRNN is essentially a Lyapunov stable dynamical system where there is no vanishing or exploding gradient in training. To demonstrate the applicability in practice, we also propose a novel implementation, namely TinyRNN, that sparsifies the transition matrices in DIRNN using weighted random permutations to reduce the model sizes. We evaluate our approach on seven benchmark datasets, and achieve state-of-the-art results. Demo code is provided in the supplementary file.