DMS-EPSRC: Asymptotic Analysis of Online Training Algorithms in Machine Learning: Recurrent, Graphical, and Deep Neural Networks

机器学习中的神经网络模型在过去十年中取得了巨大的实际成功，彻底改变了图像、文本和语音识别、工程、医学和金融等领域。用于这些复杂机器学习问题的训练算法在实践中是成功的，但它们通常是临时的。在许多情况下，数学理论尚未建立，并且有可能通过严格的分析来改进训练算法和模型。本研究的主要目的是为机器学习的几个关键领域中使用的神经网络模型的训练算法开发严格的数学分析。为广泛应用的训练算法开发和测试数学理论是保证其可靠性和在实践中发挥作用的关键。该研究项目与一个教育项目相结合，旨在帮助培养应用数学、工程、计算机科学和数据科学领域的本科生和研究生，探索训练算法、算法分析和算法性能。该项目涉及开发新的数学方法，通过严格推导McKean-Vlasov型平均场极限、中心极限定理、统计缩放极限和大偏差原理，对前馈、循环和图神经网络的训练算法进行严格分析。这将通过利用随机分析和弱收敛理论的方法来研究在线的渐近性，随机训练算法和神经网络模型随着隐藏单元的数量变得很大来实现。该项目还涉及在数学分析与参数初始化、超参数选择、优化/训练算法设计和模型选择之间建立直接联系。除了证明重要的神经网络训练算法的收敛理论外，这项研究将在机器学习之外引起人们的兴趣，因为它将研究一组具有新颖和数学挑战性特征的新平均场问题，使其方法论引起包括数学生物学，物理学和工程学在内的其他领域的兴趣。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Neural network models in machine learning have achieved immense practical success over the past decade, revolutionizing fields such as image, text, and speech recognition, engineering, medicine, and finance. The training algorithms used for these complex machine learning problems are successful in practice, but they are often ad hoc. Mathematical theory is yet to be established in many cases and there is the potential to improve training algorithms and models via rigorous analysis. The primary purpose of this research is to develop a rigorous mathematical analysis for the training algorithms for neural network models used in several key areas of machine learning. Developing and testing mathematical theory for widely-used training algorithms is crucial for ensuring their reliability and guaranteeing their performance in practice. This research project is integrated with an educational program that is designed to help in the training of undergraduate and graduate students in applied mathematics, engineering, computer science and data science in the exploration of training algorithms, their analysis and their performance. This project involves the development of new mathematical approaches for a rigorous analysis for the training algorithms for feedforward, recurrent and graph neural networks by rigorously deriving McKean-Vlasov type of mean field limits, central limit theorems, statistical scaling limits and large deviation principles. This will be achieved by leveraging methods from stochastic analysis and weak convergence theory to study the asymptotics of online, stochastic training algorithms and neural network models as the number of hidden units becomes large. The project also involves making direct connections between the mathematical analyses and parameter initialization, hyperparameter selection, the design of optimization/training algorithms, and model selection. In addition to proving convergence theory for important neural network training algorithms, the research will be of interest outside of machine learning as it will study a new set of mean-field problems with novel and mathematically challenging features, making the methodology of interest to other fields including mathematical biology, physics and engineering.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.