Wasserstein-Type Gradient Flow via Propagation by Chaos for a Continuous Formulation of a Shallow Neural Network

在过去的几十年里，统计学、数据分析和人工智能等科学学科经历了突飞猛进的进步和流行。这种增长可以归功于机器学习优化框架的发现，这是一种简单、可扩展和可参数化的数据内插方法。它的应用范围广泛，包括金融、计算机编程、3D图形、数据分析、医疗保健、流行病学、法律、城市研究、动物行为等。尽管几乎每个人都意识到，这些算法在我们的日常生活中变得越来越相互交织和无处不在，但相当违反直觉的是，我们还没有开发出一个令人满意的数学模型来描述这些机器的行为，而且还处于研究的早期阶段。这反映在我们将这些算法称为黑盒，因此，研究机器学习框架和梯度下降算法势在必行。为了优化这些机器的性能和效率，我们必须了解它们的内部工作原理。这将使我们能够用更少的计算资源和数据来调整这些机器以产生更好的结果。我们专注于利用单层神经网络来逼近数据集的具体问题，利用梯度下降算法进行均方误差和正则化。单层神经网络由有限个结点组成，每个结点由一个参数和一个权值参数化。今天，分析这一问题最广泛使用的数学框架之一是采用单层神经网络的平均场描述，并研究关于平方瓦瑟斯坦度量的梯度流的适定性和性质。然而，这种表示包含冗余，在某种意义上，神经网络不随节点权重的变化而变化。有许多证据表明，在梯度下降算法过程中，节点权重的方差将单调减小。特别是，权参数的初始零方差将在整个梯度流中传播，也就是说，我们最终得到一条所谓的Young度量曲线。这个项目的目的是建立一个严格的数学证明来证明所述猜想。此外，这个新的框架可能会为我们提供一些缺失的工具来确定梯度下降算法的定量收敛界或渐近极限的性质，这两个方面仍然知之甚少。此外，该框架还为我们提供了一个独特的机会来分析表示为参数函数的单层神经网络的权值的演化和规律性。该项目是新颖的，因为没有已知的来源来研究关于梯度下降方程的Young测度的几何。此外，这项研究将使我们对梯度流动方程平均场解的几何形状有更多的了解。

Scientific disciplines such as statistics, data analysis and artificial intelligence have witnessed a sudden surge of progress and popularity during the last few decades. This growth can be attributed to the discovery of the machine learning optimization framework, which is a simple, scalable and parameterizable method of interpolating data. Its applications span a wide area of industry including finance, computer programming, 3D graphics, data analysis, health care, epidemiology, law, urban study, animal behaviour and many more. Even though almost everyone is aware that these algorithms are becoming more and more intertwined and ubiquitous in our daily life, rather counterintuitively, we have yet to develop a satisfactory mathematical model to describe the behaviour of these machines and are still at the early stages of research. This is reflected in the fact we call these algorithms "black boxes".It is therefore imperative that we research the machine learning framework and the gradient descent algorithm. To optimize the performance and efficiency of these machines, we must gain an understanding of how they work internally. This will allow us to adjust these machines to produce better results with less computational resources and data.We focus on the specific problem of approximating a dataset with a single layer neural network with respect to the mean square error and some regularization by means of the gradient descent algorithm. A single layer neural network is comprised of finite nodes, each parameterised by a parameter and a weight. Today, one of the most widely used mathematical frameworks to analyse this problem is to employ a mean-field description of the single layer neural network and to investigate the well-posedness and properties of the gradient flow with respect to the square Wasserstein metric. However, this representation contains redundancy, in the sense that the neural network is invariant to the variance of the weight of a node. There is much evidence that suggests that the variance in the weight of a node will monotonically decrease during the gradient descent algorithm. In particular an initial zero variance in the weight parameter will be propagated throughout the entire gradient flow, that is we end up with a curve of so-called Young measures.The aim of this project is to establish a rigorous mathematical justification to prove the stated conjecture. Moreover, this new framework may provide us with the missing tools to determine quantitative convergence bounds or properties of the asymptotic limit of the gradient descent algorithm, both of which are still poorly understood. Additionally, this framework gives us the unique opportunity to also analyse the properties in evolution and regularity of the weights of the single layer neural network expressed as a function of the parameters.The project is novel, in the sense that there are no known sources that study the geometry of Young measures with respect to the gradient descent equation. Furthermore, this research will give us more insight in the geometry of mean-field solutions to the gradient flow equation.