风险敏感的马氏决策过程与强化学习及其应用

本课题旨在基于马氏决策过程(MDP)和强化学习的理论框架，研究风险敏感的学习优化理论。该类问题的系统累积回报值本质是一个随机变量，传统MDP理论只优化其数学期望，忽视了方差和概率分布信息，本课题主要研究考虑方差约束的多目标MDP优化问题以及考虑概率准则的MDP优化问题，并在此基础上研究数据驱动的强化学习优化算法来有效降低系统收益的风险，完善MDP和强化学习在风险相关性能指标下的理论体系；进一步将上述方法应用于研究金融工程中的资产组合风险管理优化问题和能源互联网中的风储联合出力波动性优化问题，通过方差和概率指标来刻画金融风险和电网安全。本课题的主要学术挑战是处理现有动态决策优化理论在优化风险指标时所面临的本质困难：风险指标的非线性和不可加性使得决策问题不符合标准MDP模型，经典动态规划的一致性选择原则和Bellman最优性方程不再成立。本课题是一个深度交叉的研究前沿，有望取得较好的研究成果。

This proposal aims to study the risk-sensitive stochastic learning and optimization theory, based on the fundamental framework of Markov decision processes (MDPs) and reinforcement learning. The accumulated reward of such stochastic dynamic systems is a random variable. However, many traditional theories only focus on the optimization of its expectation, while ignoring the variance and probabilities. We will first study the variance-related multi-objective MDPs and the probability criterion for MDPs. Based on these risk-sensitive MDP theory, we will further study to develop data-driven reinforcement learning algorithms to efficiently reduce the risk related criterion of stochastic systems, such that a comprehensive framework for risk-sensitive MDP theory and reinforcement learning can be built up. Furthermore, we will apply the risk-sensitive learning and optimization theory to solve the practical problems in financial engineering and energy Internet, including the portfolio management problem for asset returns and risks, and the fluctuation reduction of power output of the renewables and energy storage systems. We use the variance criterion and the probability criterion to quantify the metrics of financial risk and grid safety. The main challenge of the classical dynamic programming theory when applied to risk-related criteria is that the risk-related function is usually nonlinear and non-additive, which induces that the optimization problem does not fit a standard MDP model. The principle of consistent choice of dynamic programming and the Bellman’s optimality equation do not hold any more for such risk-aware MDP problems. Therefore, the research on stochastic optimization problems under risk-related criteria is a young and promising multi-disciplinary area that can potentially produce fruitful research results.