基于高阶统计量的循环神经网络梯度问题形成机理及其抑制方法研究

该项目拟研究基于高阶统计量的循环神经网络（RNN）梯度问题形成机理及其抑制方法。由于RNN模型时间延拓和隐含层权复用的结构特点，模型易出现梯度消失/爆炸现象和权重分布异化现象。实验发现，不同于序贯模型，RNN模型在训练过程中存在低阶统计量约束的权重正则不足以有效抑制梯度爆炸/消失的问题。为有效解决梯度问题，该课题拟研究引入高阶统计量，分析模型权重的高阶原点矩、高阶中心距等统计量对梯度问题抑制作用，探究权重分布异化与梯度问题的关联性，提出面向RNN类模型梯度问题形成机理的高阶统计量分析理论，设计一类新型通用的RNN梯度抑制方法，并且拓展该方法在语音识别、视频信号处理等领域的应用。

The project intends to investigate the formation mechanism and suppression methods of the recurrent neural network (RNN) gradient problem based on high-order statistics. Due to the structural characteristics of time extension and hidden layer weight reuse, the RNN model is prone to gradient disappearance/explosion and weight distribution alienation. Experiments show that, unlike the sequential model, the weight regularity constrained by low-order statistics in the training process of the RNN model is not effectively suppress the problem of gradient explosion/disappearance. In order to effectively solve the gradient problem, this project intends to study high-order statistics, analyzes the inhibitory effect of the high-order origin moments and high-order center distances of the weights on the gradient problem, and explore the correlation between the weight distribution alienation and the gradient problem. Facing the high-order statistical analysis theory of the formation mechanism of the gradient problems in RNN, design a new type of general gradient suppression method, and expand the application of this method in speech recognition, video signal processing and other fields.