Stochastic optimal control constrained by costly observations

随机控制是一种将随机效应纳入决策的重要工具。它作为一种数学模型在应用中是有用的，并且经常被用于数学金融和资源管理领域。我们项目的主要目的是通过分析、数值和计算方法的结合，对随机控制变量进行数学分析，这些变量涉及昂贵的观察。我们的模型假设访问底层状态需要严格的正成本。这相当于一个优化问题，涉及成本和信息之间的权衡。尽管这属于部分信息设置，但它与过滤问题不同，过滤问题中用户可以随时访问底层状态的函数。在维护、环境恢复和医院治疗问题的情况下，通常可以看到成本和信息之间的权衡，在这些情况下，测量或记录是昂贵的，并且不能连续观察。它也可以被看作是探索与开发的一种变体，这是一个在强盗理论和强化学习研究领域普遍存在的主题。我们发现探索上述方面的文献数量有限，数学框架尚未全面建立。在我们的工作中，我们首先考虑一个离散时间马尔可夫链模型作为起点，并对一个玩具问题提供了一个简单的直觉分析。马尔可夫链的构造为连续情况下的数值逼近提供了一种可能的离散化方案。然后，我们转向一般的连续时间设置，通过动态规划原理推导出值函数的变分PDE。这种方法与经典的完全信息情况类似，我们希望在这两种情况之间找到相似之处。特别是，我们期望变分PDE在观测成本趋于零时收敛于经典HJB方程，我们希望建立这种收敛速度。由于观测成本为正，变分偏微分方程中存在额外的积分项，这给数值求解偏微分方程增加了一层复杂性。寻找一种计算效率高的数值格式将是该项目未来的重点之一。特别是神经网络的使用可以提供克服维度诅咒的潜在途径。进一步的调查途径可能涉及分析模型中的其他约束，例如纳入时间延迟执行。该项目属于EPSRC数学分析、数值分析、统计与应用概率和运筹学研究领域。

Stochastic control is an important tool that incorporates the effect of randomness in decision-making. It is useful as a mathematical model in applications and is often employed in areas of mathematical finance and resource management. The main aim of our project is the mathematical analysis of variants of stochastic control that involves costly observations, through a combination of analytical, numerical and computational approaches. Our model assumes that access to the underlying state requires a strictly positive cost. This amounts to an optimisation problem that involves a trade off between cost and information. Although this falls under the partial information setting, this differs from the filtering problem, where a function of the underlying state is accessible to the user at all times. The trade off between cost and information can be commonly seen in instances of maintenance, environmental restoration and hospital treatment problems, where measurements or records are expensive and are not continuously observed. It can also be seen as a variant of exploration versus exploitation, a theme which is prevalent in research areas of bandit theory and reinforcement learning. We find the amount of literature exploring the above aspects limited, and the mathematical framework has yet to be established in full generality. In our work we first consider a discrete-time Markov chain model as a starting point and provide a simple analysis on a toy problem for intuition. The Markov chain construction provides us with a possible discretisation scheme for numerical approximation in the continuous case. We then move on to the general continuous-time setting, deriving a variational PDE for the value function via the dynamic programming principle. This approach is in analogy to the classical full information case and we hope to draw parallels between the two cases. In particular we expect the variational PDE to converge towards the classical HJB equation as the observation cost tends towards zero and we would like to establish this rate of convergence. Due to the positive observation cost, extra integral terms are present in the variational PDE which adds a layer of complexity when numerically solving for the PDE. The search of a computationally efficient numerical scheme will be one of the focuses of the project in the future. In particular the use of neural networks could provide a potential pathway to overcoming the curse of dimensionality. Further avenues of investigation could involve analysis of additional constraints in the model, for example the incorporation of time-delayed executions. This project falls within the EPSRC Mathematics Analysis, Numerical Analysis, Statistics and Applied Probability, and Operational Research research areas.