Deep Neural Networks for Solving Non-Markov Optimization Problems

机器学习(ML)方法作为人工智能和数据科学中解决大规模优化问题的一组非常有效的工具，最近得到了相当多的关注。在这些方法中的许多方法中存在的神经网络体系结构具有一些通用的近似性质，允许用户应用软件工具，而只需对数据进行最少的预处理或根据问题的规范来定制算法。然而，在ML运行良好的大多数情况下，数学分析还没有为基本问题提供令人满意的答案：为什么机器学习方法对解决这个问题如此有效？这个项目的目的是研究ML方法如何应用于求解非马尔可夫动态规划(DPS)，并针对这一领域的一些具体问题回答这个基本问题。研究生参与了该项目的研究，研究人员分析了一种求解非马尔可夫DPS的新方法，该方法通过训练神经网络系统来获得策略逼近函数。其主要思想类似于最近发展的求解高维倒向随机微分方程(BSDE)的方法，其中所谓的深度BSDE求解器学习高维输入的函数来逼近马尔可夫DP的最优控制。这个项目的目标不同于其他工作，因为重点是非马尔可夫DPS和来自路径依赖的障碍。深入探讨的问题包括蒙特卡罗粒子方法生成的训练集所需的精度水平，以及隐式正则化在TensorFlow算法中所起的作用。该项目还考虑了更传统的理论概念，这些概念可以证明该方法的一般有效性，例如连续函数的Kolmogorov-Arnold表示和神经网络中使用的Sigmoid函数的类型。这些结果有助于更好地从理论上理解深度BSDE应用于具有非线性滤波的DPS时背后的数学原理，并有助于回答有关该方法的重要问题--S有效性。研究生参与该项目的研究。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Machine learning (ML) methods have recently gained considerable attention as a set of tools that are very effective for solving large-scale optimization problems in artificial intelligence and data science. The neural network architecture that is present in many of these methods has some universal approximation properties that allow users to apply software tools with minimal preprocessing of data or tailoring of algorithms to the specifications of the problem. However, in most instances where ML works well, mathematical analysis does not yet offer a satisfactory answer to the fundamental question: Why is a machine learning method so effective for solving this problem? The aim of this project is to investigate how ML methods can be applied to solving non-Markov dynamic programs (DPs), and to answer this fundamental question for some specific problems in this area. Graduate students participate in the research of the project.The investigator analyzes a new method for solving non-Markov DPs, wherein a policy-approximation function is obtained by training a system of neural networks. The main idea is similar to recently-developed methods for solving high-dimensional backward stochastic differential equations (BSDEs), wherein the so-called Deep BSDE Solver learns a function of a high-dimensional input to approximate the optimal control for a Markovian DP. The aims of this project differ from those of other work because the focus is non-Markov DPs and the hurdles that come from path dependence. Issues that are explored in depth include the level of accuracy needed in the training set generated by a Monte Carlo particle method, and the role that implicit regularization plays in the TensorFlow algorithms. The project also considers more conventional theoretical concepts that may provide proof of this method's general effectiveness, such as the Kolmogorov-Arnold representation for continuous functions and the types of sigmoidal functions used in neural networks. The results contribute to an improved theoretical understanding of the mathematics behind Deep BSDE when applied to DPs with nonlinear filtering, and help answer important questions regarding the method?s effectiveness. Graduate students participate in the research of the project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.