Reinforcement Learning in Low-Rank MDPs with Density Features

Reinforcement Learning in Low-Rank MDPs with Density Features
复制标题

DOI：
10.48550/arxiv.2302.02252
复制
发表时间：
2023-02
期刊：
ArXiv
影响因子：
--
通讯作者：
Audrey Huang;Jinglin Chen;Nan Jiang
Audrey Huang;Jinglin Chen;Nan Jiang
中科院分区：
其他
文献类型：
--
作者：
Audrey Huang;Jinglin Chen;Nan Jiang

文献摘要

相似文献

具有低秩转换的 MDP（即转换矩阵可以分解为左右两个矩阵的乘积）是一种高度具有代表性的结构，可以实现易于处理的学习。左矩阵可以实现基于价值的学习的表达函数逼近，并且已经得到了广泛的研究。在这项工作中，我们转而研究具有密度特征（即正确的矩阵）的样本有效学习，这会产生强大的状态占用分布模型。此设置不仅有助于在强化学习中利用无监督学习，而且还支持凸强化学习的插件解决方案。在离线设置中，我们提出了一种用于非策略占用率估计的算法，可以处理非探索性数据。以此作为子程序，我们进一步设计了一种在线算法，以逐级的方式构建探索性数据分布。作为一个核心技术挑战，占用率估计的加性误差与数据覆盖范围的乘法定义不兼容。在缺乏诸如可达性之类的强假设的情况下，这种不兼容性很容易导致指数错误激增，我们通过新颖的技术工具克服了这一点。当密度特征未知并且必须从指数级大的候选集中学习时，我们的结果也很容易扩展到表示学习设置。
MDPs with low-rank transitions -- that is, the transition matrix can be factored into the product of two matrices, left and right -- is a highly representative structure that enables tractable learning. The left matrix enables expressive function approximation for value-based learning and has been studied extensively. In this work, we instead investigate sample-efficient learning with density features, i.e., the right matrix, which induce powerful models for state-occupancy distributions. This setting not only sheds light on leveraging unsupervised learning in RL, but also enables plug-in solutions for convex RL. In the offline setting, we propose an algorithm for off-policy estimation of occupancies that can handle non-exploratory data. Using this as a subroutine, we further devise an online algorithm that constructs exploratory data distributions in a level-by-level manner. As a central technical challenge, the additive error of occupancy estimation is incompatible with the multiplicative definition of data coverage. In the absence of strong assumptions like reachability, this incompatibility easily leads to exponential error blow-up, which we overcome via novel technical tools. Our results also readily extend to the representation learning setting, when the density features are unknown and must be learned from an exponentially large candidate set.