Stochastic Optimal Control based on Gaussian Processes Regression

在随机控制中，在统计建模的不确定性下，连续域的最优决策通常通过动态规划（DP）来解决。目标在于找到将控制器可用信息映射到控制输入的策略，从而使性能标准（通常根据成本定义）得到优化。通常，使用非线性过滤方法，这些信息被压缩成表示待控制系统状态估计的概率分布，策略将这些分布映射到控制输入。不幸的是，除了少数非常特殊的情况，DP是难以治疗的。因此，近似但易于处理的方法是令人感兴趣的。其中一种方法是基于点的值迭代算法，其中每个点都是一个概率分布。在这种方法中，控制器维护一组有代表性的状态估计的最优成本，而不是像在经典DP中那样尝试维护所有状态估计的成本这一不可能完成的任务。然后，它使用这些信息在决策所需的状态估计下获得最优成本的近似值。正如我们所看到的，基于点的值迭代需要在一般概率分布上定义的函数的近似方法。然而，最先进的方法要么限制可能状态估计的类别，要么假设有限的控制输入和测量集。虽然存在连续控制输入和测量的变通方法，但它们通常需要额外的近似值。基于这个原因，我们提出了一种基于高斯过程（GP）回归的非线性动力系统随机控制的新方法，该方法具有连续状态、控制输入和测量。经典GP回归只允许确定性的向量值输入。基于这个原因，我们提出了一种新的GP框架扩展到以概率分布形式给出的输入。通过这样做，我们将GP框架扩展到无限维输入。我们的方法是基于定义协方差函数的思想，协方差函数根据作为GP输入的概率分布之间的距离来确定GP。在项目过程中，我们计划为一般概率分布上定义的gp开发一个坚实的框架，并推导出使用此类gp计算策略的随机控制算法。我们相信该项目将对随机控制的研究做出实质性的贡献。此外，本文提出的用概率分布给出输入来定义gp的想法也可以用于机器学习研究，以便在概率分布上推导出其他非参数贝叶斯回归和分类方法。

In stochastic control, optimal decision making in continuous domains under statistically modeled uncertainty is usually addressed via Dynamic Programming (DP). The goal consists in finding policies that map the information available to the controller to a control input in such a way that a performance criterion, often defined in terms of costs, is optimized. Usually, using nonlinear filtering methods, this information is condensed into a probability distribution that represents the state estimate of the system to be controlled, and the policies map these distributions to control inputs.Unfortunately, DP is intractable except in a few very special cases. Therefore, approximate but tractable approaches are of interest. One such approach is the point-based value iteration algorithm, where each point is a probability distribution. In this approach, the controller maintains the optimal costs for a set of representative state estimates instead of trying the impossible task of maintaining the costs for all state estimates as it would be required in classical DP. Then, it uses this information in order to obtain an approximation of the optimal costs at a state estimate that is needed for decision making. As we see, point-based value iteration requires approximation methods for functions defined over general probability distributions. However, state-of-the-art approaches either restrict the class of possible state estimates or assume finite sets of control inputs and measurements. Although workarounds for continuous control inputs and measurements exist, they usually require additional approximations. For this reason, we propose a novel approach to stochastic control of nonlinear dynamical systems with continuous states, control inputs, and measurements that is based on Gaussian Process (GP) regression. Classical GP regression only allows for deterministic vector-valued inputs. For this reason, we propose a novel extension of the GP framework to inputs given in form of probability distributions. By doing so, we extend the GP framework to infinite-dimensional inputs. Our approach is based on the idea to define the covariance functions that determine the GP in terms of the distance between the probability distributions provided as inputs to the GP.In the course of the project, we plan to develop a solid framework for GPs defined over general probability distributions and to derive stochastic control algorithms that use such GPs to compute the policy. We believe that the proposed project will substantially contribute to research on stochastic control. Furthermore, the presented idea for defining GPs with inputs given in terms of probability distributions can also be used in machine learning research in order to derive other non-parametric Bayesian regression and classification methods over probability distributions.