Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare-猫眼课题宝

DOI：

10.48550/arxiv.2305.01738

发表时间：

2023-05

期刊：

ArXiv

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Shengpu Tang;Maggie Makar;M. Sjoding;F. Doshi-Velez;J. Wiens

许多强化学习（RL）应用程序具有组合动作空间，其中每个动作都是子动作的组合。标准的RL方法忽略了这种固有的因子分解结构，导致可能无法对很少观察到的子动作组合进行有意义的推断;这对于数据可能有限的离线设置尤其有问题。在这项工作中，我们提出了一种形式的线性Q-函数分解诱导因子的行动空间。我们研究了我们的方法的理论特性，确定的情况下，它是保证导致零偏差时，用于近似的Q函数。在有理论保证的制度之外，我们表明我们的方法仍然是有用的，因为它可以提高样本效率，而不一定牺牲政策最优性，使我们能够实现更好的偏差-方差权衡。在使用模拟器和医疗保健激励的真实世界数据集的几个离线RL问题中，我们证明了将因子动作空间纳入基于值的RL可以产生更好的性能策略。我们的方法可以帮助智能体在将RL应用于观测数据集时，在状态-动作空间的未充分探索区域内做出更准确的推断。

Many reinforcement learning (RL) applications have combinatorial action spaces, where each action is a composition of sub-actions. A standard RL approach ignores this inherent factorization structure, resulting in a potential failure to make meaningful inferences about rarely observed sub-action combinations; this is particularly problematic for offline settings, where data may be limited. In this work, we propose a form of linear Q-function decomposition induced by factored action spaces. We study the theoretical properties of our approach, identifying scenarios where it is guaranteed to lead to zero bias when used to approximate the Q-function. Outside the regimes with theoretical guarantees, we show that our approach can still be useful because it leads to better sample efficiency without necessarily sacrificing policy optimality, allowing us to achieve a better bias-variance trade-off. Across several offline RL problems using simulators and real-world datasets motivated by healthcare, we demonstrate that incorporating factored action spaces into value-based RL can result in better-performing policies. Our approach can help an agent make more accurate inferences within underexplored regions of the state-action space when applying RL to observational datasets.

Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare

文献摘要

相似文献