Stochastic optimal control constrained by costly observations

随机控制是一种重要的工具，它将随机性的影响纳入决策。它在应用中作为数学模型很有用，并且经常在数学金融和资源管理领域中使用。我们项目的主要目的是通过分析，数值和计算方法的结合，对涉及昂贵观测的随机控制变体进行数学分析。我们的模型假设进入底层状态需要严格的正成本。这相当于一个优化问题，涉及成本和信息之间的权衡。虽然这福尔斯属于部分信息设置，但这与过滤问题不同，在过滤问题中，用户始终可以访问底层状态的函数。在维护、环境恢复和医院治疗问题的情况下，通常可以看到成本和信息之间的权衡，在这些情况下，测量或记录是昂贵的，并且没有持续观察。它也可以被看作是探索与剥削的一种变体，这是一个在强盗理论和强化学习研究领域中普遍存在的主题。 我们发现，探讨上述方面的文献数量有限，数学框架尚未建立在充分的普遍性。在我们的工作中，我们首先考虑一个离散时间马尔可夫链模型作为出发点，并提供一个简单的分析玩具问题的直觉。马尔可夫链的建设为我们提供了一个可能的离散化方案的数值逼近连续的情况下。然后，我们移动到一般的连续时间设置，通过动态规划原理导出值函数的变分偏微分方程。这种方法类似于经典的完全信息的情况下，我们希望这两种情况之间的相似之处。特别是，我们希望变分偏微分方程收敛到经典的HJB方程的观测成本趋于零，我们想建立这种收敛速度。由于正的观测成本，额外的积分项存在于变分偏微分方程中，这增加了一层的复杂性时，数值求解偏微分方程。寻找一种计算效率高的数值格式将是该项目未来的重点之一。特别是神经网络的使用可以提供一个潜在的途径来克服维数灾难。进一步的调查途径可包括分析该模式中的其他限制因素，例如纳入延迟处决。该项目属于EPSRC数学分析、数值分析、统计和应用概率以及运筹学研究领域的福尔斯。

Stochastic control is an important tool that incorporates the effect of randomness in decision-making. It is useful as a mathematical model in applications and is often employed in areas of mathematical finance and resource management. The main aim of our project is the mathematical analysis of variants of stochastic control that involves costly observations, through a combination of analytical, numerical and computational approaches. Our model assumes that access to the underlying state requires a strictly positive cost. This amounts to an optimisation problem that involves a trade off between cost and information. Although this falls under the partial information setting, this differs from the filtering problem, where a function of the underlying state is accessible to the user at all times. The trade off between cost and information can be commonly seen in instances of maintenance, environmental restoration and hospital treatment problems, where measurements or records are expensive and are not continuously observed. It can also be seen as a variant of exploration versus exploitation, a theme which is prevalent in research areas of bandit theory and reinforcement learning. We find the amount of literature exploring the above aspects limited, and the mathematical framework has yet to be established in full generality. In our work we first consider a discrete-time Markov chain model as a starting point and provide a simple analysis on a toy problem for intuition. The Markov chain construction provides us with a possible discretisation scheme for numerical approximation in the continuous case. We then move on to the general continuous-time setting, deriving a variational PDE for the value function via the dynamic programming principle. This approach is in analogy to the classical full information case and we hope to draw parallels between the two cases. In particular we expect the variational PDE to converge towards the classical HJB equation as the observation cost tends towards zero and we would like to establish this rate of convergence. Due to the positive observation cost, extra integral terms are present in the variational PDE which adds a layer of complexity when numerically solving for the PDE. The search of a computationally efficient numerical scheme will be one of the focuses of the project in the future. In particular the use of neural networks could provide a potential pathway to overcoming the curse of dimensionality. Further avenues of investigation could involve analysis of additional constraints in the model, for example the incorporation of time-delayed executions. This project falls within the EPSRC Mathematics Analysis, Numerical Analysis, Statistics and Applied Probability, and Operational Research research areas.