Stochastic optimal control constrained by costly observations

随机控制是决策中综合考虑随机性影响的重要工具。它在应用程序中作为数学模型很有用，经常被用于数学金融和资源管理领域。我们项目的主要目标是通过分析、数值和计算方法的组合，对涉及昂贵观测的随机控制变量进行数学分析。我们的模型假设，进入基础国家需要严格的正成本。这相当于一个优化问题，涉及到成本和信息之间的权衡。尽管这属于部分信息设置，但这不同于过滤问题，在过滤问题中，用户始终可以访问底层状态的函数。在维护、环境恢复和医院治疗问题中，通常可以看到成本和信息之间的权衡，在这些情况下，测量或记录是昂贵的，并且不是连续观察的。它也可以被视为探索与剥削的变体，这是土匪理论和强化学习研究领域中的一个流行主题。我们发现研究上述方面的文献数量有限，数学框架还没有完全建立起来。在我们的工作中，我们首先考虑一个离散时间的马尔可夫链模型作为起点，并对一个玩具问题进行简单的直观分析。马尔可夫链的构造为连续情况下的数值逼近提供了一种可能的离散化方案。然后我们继续到一般的连续时间设置，通过动态规划原理推导出值函数的变分偏微分方程。这种方法类似于经典的完全信息情况，我们希望将这两种情况进行比较。特别地，我们期望当观测成本趋于零时，变分偏微分方程会收敛到经典的HJB方程，我们想要建立这个收敛速度。由于观测成本为正，在变分偏微分方程组中引入了额外的积分项，这增加了偏微分方程组数值求解的复杂性。寻找一种计算效率高的数值格式将是今后该项目的重点之一。特别是，神经网络的使用可以提供一条克服维度诅咒的潜在途径。进一步的调查途径可能涉及分析模型中的其他制约因素，例如纳入延时执行。该项目属于EPSRC数学分析、数值分析、统计和应用概率以及运筹学研究领域。

Stochastic control is an important tool that incorporates the effect of randomness in decision-making. It is useful as a mathematical model in applications and is often employed in areas of mathematical finance and resource management. The main aim of our project is the mathematical analysis of variants of stochastic control that involves costly observations, through a combination of analytical, numerical and computational approaches. Our model assumes that access to the underlying state requires a strictly positive cost. This amounts to an optimisation problem that involves a trade off between cost and information. Although this falls under the partial information setting, this differs from the filtering problem, where a function of the underlying state is accessible to the user at all times. The trade off between cost and information can be commonly seen in instances of maintenance, environmental restoration and hospital treatment problems, where measurements or records are expensive and are not continuously observed. It can also be seen as a variant of exploration versus exploitation, a theme which is prevalent in research areas of bandit theory and reinforcement learning. We find the amount of literature exploring the above aspects limited, and the mathematical framework has yet to be established in full generality. In our work we first consider a discrete-time Markov chain model as a starting point and provide a simple analysis on a toy problem for intuition. The Markov chain construction provides us with a possible discretisation scheme for numerical approximation in the continuous case. We then move on to the general continuous-time setting, deriving a variational PDE for the value function via the dynamic programming principle. This approach is in analogy to the classical full information case and we hope to draw parallels between the two cases. In particular we expect the variational PDE to converge towards the classical HJB equation as the observation cost tends towards zero and we would like to establish this rate of convergence. Due to the positive observation cost, extra integral terms are present in the variational PDE which adds a layer of complexity when numerically solving for the PDE. The search of a computationally efficient numerical scheme will be one of the focuses of the project in the future. In particular the use of neural networks could provide a potential pathway to overcoming the curse of dimensionality. Further avenues of investigation could involve analysis of additional constraints in the model, for example the incorporation of time-delayed executions. This project falls within the EPSRC Mathematics Analysis, Numerical Analysis, Statistics and Applied Probability, and Operational Research research areas.