Universal Off-Policy Evaluation
Universal Off-Policy Evaluation
复制标题
DOI：
--
复制
发表时间：
2021-04
期刊：
影响因子：
--
通讯作者：
Yash Chandak;S. Niekum;Bruno C. da Silva;E. Learned-Miller;E. Brunskill;P. Thomas
中科院分区：
文献类型：
--
作者：
Yash Chandak;S. Niekum;Bruno C. da Silva;E. Learned-Miller;E. Brunskill;P. Thomas
当面临连续的决策问题时，能够预测如果使用新策略做出决策会发生什么，这通常是有用的。这些预测通常必须基于根据以前使用的一些决策规则收集的数据。许多以前的方法允许这种非策略(或反事实)的估计被称为回报的业绩指标的期望值。在这篇文章中，我们向一个通用的非政策估计(UNO)迈出了第一步--它为收益分布的任何参数提供非政策估计和高置信限。我们使用UNO来估计和同时界定均值、方差、分位数/中位数、分位数间范围、CVaR和整个累积收益分布。最后，我们还讨论了Uno在各种情况下的适用性，包括完全可观测、部分可观测(即，在有未观测混杂的情况下)、马尔可夫、非马尔可夫、平稳、光滑非平稳和离散分布位移。
When faced with sequential decision-making problems, it is often useful to be able to predict what would happen if decisions were made using a new policy. Those predictions must often be based on data collected under some previously used decision-making rule. Many previous methods enable such off-policy (or counterfactual) estimation of the expected value of a performance measure called the return. In this paper, we take the first steps towards a universal off-policy estimator (UnO) -- one that provides off-policy estimates and high-confidence bounds for any parameter of the return distribution. We use UnO for estimating and simultaneously bounding the mean, variance, quantiles/median, inter-quantile range, CVaR, and the entire cumulative distribution of returns. Finally, we also discuss Uno's applicability in various settings, including fully observable, partially observable (i.e., with unobserved confounders), Markovian, non-Markovian, stationary, smoothly non-stationary, and discrete distribution shifts.