Robust, risk-sensitive, and data-driven control of markov decision processes
Robust, risk-sensitive, and data-driven control of markov decision processes
复制标题
对马尔可夫决策过程进行稳健、风险敏感且数据驱动的控制
DOI：
--
复制
发表时间：
2007
期刊：
影响因子：
--
通讯作者：
Y. L. Tallec
中科院分区：
文献类型：
--
作者：
D. Simester;J. Tsitsiklis;Y. L. Tallec
马尔可夫决策过程（MDP）是不确定条件下的序贯决策问题的模型。它们已被广泛研究和应用。然而，有两个主要的障碍，仍然阻碍了MDP的适用性，许多更实际的决策问题：（1）决策者往往缺乏一个可靠的MDP模型。由于动态规划得到的结果是敏感的假设MDP模型，其相关性受到挑战的模型的不确定性。(2)动态规划（处理预期性能）的结构和计算结果已经扩展到只有有限的成功，以适应风险敏感的决策者。
在这篇论文中，我们研究了两种处理不确定MDP的方法，并将不确定MDP的鲁棒控制和动力系统的风险敏感控制联系起来。
第一，方法假设一个模型的不确定性和制定的不确定MDP的控制作为（模型）不确定性下的决策问题。我们确定，大多数公式至少是NP-难的，因此遭受“不确定性的诅咒”。'
具有矩形不确定集的MDP的最坏情况控制等价于控制器与自然之间的零和博弈。这种游戏的结构和计算结果，使这个配方吸引人。通过对不太可能的参数增加惩罚，我们扩展了不确定MDP的最坏情况控制的制定，并减轻其保守性。我们表现出惩罚最坏情况下的控制不确定的MDP矩形的不确定性和最小化的马尔可夫动态一致的凸风险测度的样本成本之间的对偶。这个概念的风险多期决策，包括一个新的马尔可夫财产，我们介绍和激励理想的属性。这种马尔可夫性质对于建立最小化样本成本的某些风险度量与解决决策者与自然之间的某种零和马尔可夫博弈之间的等价性以及解决无限时间范围问题至关重要。
处理不确定MDP的另一种方法是直接利用观测数据，这避免了不确定性的诅咒。具体来说，我们从一个训练集估计任何给定策略的预期性能（及其相对于某些策略参数的梯度），该训练集包括在已知策略下采样的观察轨迹。我们提出了新的值（和值梯度）估计，是无偏的，具有低训练集到训练集的方差。我们希望我们的方法优于竞争的方法时，有几个系统的观察相比，底层MDP的大小，数值实验表明。（副本可从麻省理工学院图书馆，RM。14-0551，剑桥，MA 02139-4307。电话：617-253-5668;传真：617-253-1690。）
Markov Decision Processes (MDPs) model problems of sequential decision-making under uncertainty. They have been studied and applied extensively. Nonetheless, there are two major barriers that still hinder the applicability of MDPs to many more practical decision making problems: (1) The decision maker is often lacking a reliable MDP model. Since the results obtained by dynamic programming are sensitive to the assumed MDP model, their relevance is challenged by model uncertainty. (2) The structural and computational results of dynamic programming (which deals with expected performance) have been extended with only limited success to accommodate risk-sensitive decision makers.
In this thesis, we investigate two ways of dealing with uncertain MDPs and we develop a new connection between robust control of uncertain MDPs and risk-sensitive control of dynamical systems.
The first, approach assumes a model of model uncertainty and formulates the control of uncertain MDPs as a problem of decision-making under (model) uncertainty. We establish that most formulations are at least NP-hard and thus suffer from the 'curse of uncertainty.'
The worst-case control of MDPs with rectangular uncertainty sets is equivalent to a zero-sum game between the controller and nature. The structural and computational results for such games make this formulation appealing. By adding a penalty for unlikely parameters, we extend the formulation of worst-case control of uncertain MDPs and mitigate its conservativeness. We show a duality between the penalized worst-case control of uncertain MDPs with rectangular uncertainty and the minimization of a Markovian dynamically consistent convex risk measure of the sample cost. This notion of risk has desirable properties for multi-period decision making, including a new Markovian property that we introduce and motivate. This Markovian property is critical in establishing the equivalence between minimizing some risk measure of the sample cost and solving a certain zero-sum Markov game between the decision maker and nature, and to tackling infinite-horizon problems.
An alternative approach to dealing with uncertain MDPs, which avoids the curse of uncertainty, is to exploit directly observational data. Specifically, we estimate the expected performance of any given policy (and its gradient with respect to certain policy parameters) from a training set comprising observed trajectories sampled under a known policy. We propose new value (and value gradient) estimators that are unbiased and have low training set to training set variance. We expect our approach to outperform competing approaches when there are few system observations compared to the underlying MDP size, as indicated by numerical experiments. (Copies available exclusively from MIT Libraries, Rm. 14-0551, Cambridge, MA 02139-4307. Ph. 617-253-5668; Fax 617-253-1690.)