Stochastic optimal control constrained by costly observations

点击翻译按钮获取中文摘要

Stochastic control is an important tool that incorporates the effect of randomness in decision-making. It is useful as a mathematical model in applications and is often employed in areas of mathematical finance and resource management. The main aim of our project is the mathematical analysis of variants of stochastic control that involves costly observations, through a combination of analytical, numerical and computational approaches. Our model assumes that access to the underlying state requires a strictly positive cost. This amounts to an optimisation problem that involves a trade off between cost and information. Although this falls under the partial information setting, this differs from the filtering problem, where a function of the underlying state is accessible to the user at all times. The trade off between cost and information can be commonly seen in instances of maintenance, environmental restoration and hospital treatment problems, where measurements or records are expensive and are not continuously observed. It can also be seen as a variant of exploration versus exploitation, a theme which is prevalent in research areas of bandit theory and reinforcement learning. We find the amount of literature exploring the above aspects limited, and the mathematical framework has yet to be established in full generality. In our work we first consider a discrete-time Markov chain model as a starting point and provide a simple analysis on a toy problem for intuition. The Markov chain construction provides us with a possible discretisation scheme for numerical approximation in the continuous case. We then move on to the general continuous-time setting, deriving a variational PDE for the value function via the dynamic programming principle. This approach is in analogy to the classical full information case and we hope to draw parallels between the two cases. In particular we expect the variational PDE to converge towards the classical HJB equation as the observation cost tends towards zero and we would like to establish this rate of convergence. Due to the positive observation cost, extra integral terms are present in the variational PDE which adds a layer of complexity when numerically solving for the PDE. The search of a computationally efficient numerical scheme will be one of the focuses of the project in the future. In particular the use of neural networks could provide a potential pathway to overcoming the curse of dimensionality. Further avenues of investigation could involve analysis of additional constraints in the model, for example the incorporation of time-delayed executions. This project falls within the EPSRC Mathematics Analysis, Numerical Analysis, Statistics and Applied Probability, and Operational Research research areas.