The Optimal Approximation Factors in Misspecified Off-Policy Value Function Estimation
The Optimal Approximation Factors in Misspecified Off-Policy Value Function Estimation
复制标题
DOI：
10.48550/arxiv.2307.13332
复制
发表时间：
2023-07
期刊：
影响因子：
--
通讯作者：
P. Amortila;Nan Jiang;Csaba Szepesvari
中科院分区：
文献类型：
--
作者：
P. Amortila;Nan Jiang;Csaba Szepesvari
已知强化学习（RL）中的理论保证在函数近似的误规范误差方面受到乘法放大因素的影响。然而，这种\emph{近似因子}的本质——尤其是它们在给定学习问题中的最佳形式——却知之甚少。本文研究了线性离策略值函数估计中的这一问题，其中仍有许多有待解决的问题。我们在广泛的设置范围内研究近似因子，例如加权$L_2$ -范数（其中权重是离线状态分布），$L_\infty$范数，状态混叠的存在与不存在，以及状态空间的完全覆盖与部分覆盖。我们为所有这些设置建立了最优的渐近近似因子（直至常数）。特别是，我们的界限确定了$L_2(\mu)$规范的两个实例相关因素，而$L_\infty$规范只有一个实例相关因素，这表明了在错误规范下偏离策略评估的硬度。
Theoretical guarantees in reinforcement learning (RL) are known to suffer multiplicative blow-up factors with respect to the misspecification error of function approximation. Yet, the nature of such \emph{approximation factors} -- especially their optimal form in a given learning problem -- is poorly understood. In this paper we study this question in linear off-policy value function estimation, where many open questions remain. We study the approximation factor in a broad spectrum of settings, such as with the weighted $L_2$-norm (where the weighting is the offline state distribution), the $L_\infty$ norm, the presence vs. absence of state aliasing, and full vs. partial coverage of the state space. We establish the optimal asymptotic approximation factors (up to constants) for all of these settings. In particular, our bounds identify two instance-dependent factors for the $L_2(\mu)$ norm and only one for the $L_\infty$ norm, which are shown to dictate the hardness of off-policy evaluation under misspecification.