DMS-EPSRC: Asymptotic Analysis of Online Training Algorithms in Machine Learning: Recurrent, Graphical, and Deep Neural Networks

机器学习中的神经网络模型在过去十年中取得了巨大的实际成功，彻底改变了图像、文本和语音识别等领域。神经网络在科学、工程、医学和金融领域也得到了广泛的应用。特别是，使用多层神经网络的深度学习已经改变了机器学习领域。用于这些复杂机器学习问题的训练算法-尽管在实践中是成功的-通常是特设的。在许多情况下，数学理论尚未建立，并且有可能通过严格的数学分析来改进训练算法和模型。本研究的主要目的是为机器学习的几个关键领域中使用的训练算法和神经网络模型开发新的数学理论。在这个项目中的问题是由两个基本的数学问题和高度相关的应用问题的动机。为广泛使用的训练算法开发和测试数学理论对于确保其可靠性并保证其在实践中的性能至关重要。这项研究计划的成功完成将有助于机器学习社区开发新的数学方法，可广泛用于神经网络和机器学习算法的分析。该研究项目与一个教育项目相结合，将有助于培养学生在机器学习和深度学习的数学基础方面的能力。我们提出的研究将为机器学习的几个重要领域中使用的训练算法和神经网络模型进行严格的数学分析，包括：递归神经网络，强化学习，图神经网络和深度神经网络。我们的分析将描述这些算法和神经网络模型的渐近行为，因为训练步骤的数量和神经网络中隐藏单元的数量趋于无穷大。研究人员将开发新的数学方法，通过利用随机分析和弱收敛理论的方法来研究在线随机训练算法和神经网络模型的渐近性，以分析神经网络。将研究数学结果在参数初始化、超参数选择、优化/训练算法设计以及模型结构选择等方面的应用。该研究项目是高度跨学科的，利用概率，偏微分方程，大偏差理论，随机分析，优化和机器学习的方法。除了证明重要神经网络训练算法的收敛理论外，这项研究还将在机器学习之外引起更广泛的兴趣，因为它将研究一组具有新颖和数学挑战性特征的新平均场问题。

Neural network models in machine learning have achieved immense practical success over the past decade, revolutionizing fields such as image, text, and speech recognition. Neural networks have also become widely-used in science, engineering, medicine, and finance. In particular, deep learning, which uses multi-layer neural networks, has transformed the field of machine learning. The training algorithms used for these complex machine learning problems -- although successful in practice -- are often ad hoc. Mathematical theory is yet to be established in many cases, and there is the potential to improve training algorithms and models via rigorous mathematical analysis. The primary purpose of this research is to develop new mathematical theory for the training algorithms and neural network models used in several key areas of machine learning. The problems in this project are motivated by both fundamental mathematical questions and questions highly relevant to applications. Developing and testing mathematical theory for widely-used training algorithms is crucial for ensuring their reliability and guaranteeing their performance in practice. The successful conclusion of this proposal's research program will contribute to the machine learning community by developing new mathematical methods which can be broadly used for the analysis of neural networks and machine learning algorithms. This research project is integrated with an educational programme which will contribute to the training of students in the mathematical foundations of machine learning and deep learning.Our proposed research will develop a rigorous mathematical analysis for the training algorithms and neural network models used in several important areas of machine learning, including: recurrent neural networks, reinforcement learning, graph neural networks, and deep neural networks. Our analysis will characterize the asymptotic behavior of these algorithms and neural network models as the number of training steps and the number of hidden units in the neural network go to infinity. The researchers will develop new mathematical methods designed for the analysis of neural networks by leveraging methods from stochastic analysis and weak convergence theory to study the asymptotics of online, stochastic training algorithms and neural network models as the number of hidden units becomes large. Applications of the mathematical results to parameter initialization, hyperparameter selection, design of optimization/training algorithms, and the selection of model architectures will be investigated. The research project is highly interdisciplinary, leveraging methods from probability, partial differential equations, large deviations theory, stochastic analysis, optimization, and machine learning. In addition to proving convergence theory for important neural network training algorithms, the research will be of broader interest outside of machine learning as it will study a new set of mean-field problems with novel and mathematically challenging features.