Asymptotic analysis of online training algorithms in deep learning

由于神经网络能够从大型数据集中学习高维、非线性关系，因此在科学、工程和金融的各个领域取得了巨大的成功。神经网络的参数通过“训练”神经网络来校准，以使用随机梯度下降（SGD）方法最小化数据集的适当目标函数。目前对SGD方法为什么可以成功训练神经网络以及这种神经网络如何推广到新的样本外数据的数学理解仅限于神经网络具有简单架构的情况。计划研究的目标是双重的。首先，我们将为更复杂的神经网络架构发展数学理论。一个值得注意的例子是递归神经网络（rnn）家族，它包含一个隐藏状态，该状态具有前一个时间步骤的数据序列的“记忆”。分析rnn需要新的方法。内存更新很大程度上依赖于输入数据序列的分布，这将随着时间的推移而相关。另一个例子是深度强化学习算法，如演员批评神经网络算法。这些算法在数学上分析是具有挑战性的，因为它们同时学习动态和最优策略。此外，在训练过程中，数据的分布随着强化学习模型的变化而变化。在我们的分析中，我们计划研究单层和多层（深度）神经网络。其次，我们的分析将尝试研究在应用程序中实现深度学习模型的重要基础问题，包括信息如何传播（梯度消失/爆炸问题）。我们的研究将有助于深度学习的数学理论。深度学习模型的收敛和泛化理论对于保证深度学习在应用中实现的可靠性和准确性至关重要。该项目属于以下EPSRC研究领域：非线性系统，统计和应用概率，数值分析和数学科学。

Neural networks have achieved immense practical success in various fields of science, engineering and finance due to their ability to learn high-dimensional, nonlinear relationships from large datasets. The parameters of the neural network are calibrated by 'training' the neural network to minimise an appropriate objective function for a dataset using stochastic gradient descent (SGD) methods. Current mathematical understanding of why SGD methods can successfully train a neural network and how such a neural network generalizes to new out-of-sample data is limited to cases where the neural network has a simple architecture. The objectives of the planned research is twofold. First, we will develop mathematical theory for more sophisticated neural network architectures. A notable example is the family of recurrent neural networks (RNNs), which include a hidden state with the "memory" of the data sequence at previous time steps. Novel approaches are required to analyse RNNs. The memory updates depend strongly on the distribution of the input data sequence, which will be correlated across time. Another example is deep reinforcement learning algorithms such as actor-critic neural network algorithms. These algorithms are challenging to mathematically analyse since they are simultaneously learning the dynamics as well as an optimal policy. Furthermore, the distribution of the data changes as the reinforcement learning model changes during training. In our analysis, we plan to study both single-layer and multi-layer (deep) neural networks. Secondly, our analysis will attempt to study important fundamental questions for the implementation of deep learning models in applications, including how information propagates (the vanishing/exploding gradient problem). Our research will contribute to the mathematical theory of deep learning. Convergence and generalization theory for deep learning models is important to guarantee the reliability and accuracy of deep learning when implemented in applications. This project falls within the following EPSRC research areas: non-linear systems, statistics and applied probability, numerical analysis, and mathematical sciences.