A Convex Optimization Approach to Distributionally Robust Markov Decision Processes With Wasserstein Distance
A Convex Optimization Approach to Distributionally Robust Markov Decision Processes With Wasserstein Distance
复制标题
DOI：
10.1109/lcsys.2017.2711553
复制
发表时间：
2017-06
影响因子：
3
通讯作者：
Insoon Yang
中科院分区：
文献类型：
--
作者：
Insoon Yang
我们考虑的问题，构造控制策略，对分布误差的马尔可夫决策过程的模型参数是强大的。Wasserstein度量被用来模拟可容许分布的模糊集。我们证明了马尔可夫政策的存在性和最优性，并开发了基于凸优化的工具来计算和分析的政策。我们的方法，这是基于Kantorovich凸松弛和对偶原理，具有以下优点。首先，建议的对偶制定相关的贝尔曼方程解决了无限维的问题，这是固有的在其原始配方时，名义分布有一个有限的支持。其次，我们的对偶分析确定了最坏情况分布的结构，并为其构造提供了一个简单的分散方法。第三，开发了一个敏感性分析工具来量化模糊集参数对分布鲁棒策略性能的影响。我们所提出的工具的有效性是通过一个以人为中心的空调问题。
We consider the problem of constructing control policies that are robust against distribution errors in the model parameters of Markov decision processes. The Wasserstein metric is used to model the ambiguity set of admissible distributions. We prove the existence and optimality of Markov policies and develop convex optimization-based tools to compute and analyze the policies. Our methods, which are based on the Kantorovich convex relaxation and duality principle, have the following advantages. First, the proposed dual formulation of an associated Bellman equation resolves the infinite dimensionality issue that is inherent in its original formulation when the nominal distribution has a finite support. Second, our duality analysis identifies the structure of a worst-case distribution and provides a simple decentralized method for its construction. Third, a sensitivity analysis tool is developed to quantify the effect of ambiguity set parameters on the performance of distributionally robust policies. The effectiveness of our proposed tools is demonstrated through a human-centered air conditioning problem.