Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings
Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings
复制标题
DOI：
--
复制
发表时间：
2021-07
期刊：
影响因子：
--
通讯作者：
Shengpu Tang;J. Wiens
中科院分区：
文献类型：
--
作者：
Shengpu Tang;J. Wiens
强化学习（RL）可用于学习医疗保健中的治疗策略和辅助决策。然而，考虑到需要在复杂的状态/动作空间上进行泛化，函数逼近器（例如，深度神经网络）的结合需要模型选择来减少过拟合并提高部署时的策略性能。然而，用于模型选择的标准验证管道需要在实际环境中运行学习到的策略，这在医疗保健环境中通常是不可行的。在这项工作中，我们研究了离线RL的模型选择管道，该管道依赖于非策略评估（OPE）作为验证性能的代理。我们对流行的OPE方法进行了深入分析，强调了用于对一组候选策略进行排名时的额外超参数和计算需求（辅助模型的拟合/推理）。我们比较了不同的OPE方法的效用，作为学习治疗脓毒症患者的模型选择管道的一部分。在我们考虑的所有OPE方法中，拟合Q评估（FQE）始终导致最佳验证排名，但计算成本较高。为了平衡排序准确性和计算效率之间的权衡，我们提出了一种简单的两阶段方法，通过避免潜在的不必要的计算来加速模型选择。我们的工作作为离线强化学习模型选择的实用指南，可以帮助强化学习从业者使用真实世界的数据集选择策略。为了便于再现性和将来的扩展，本文附带的代码可在线获得。
Reinforcement learning (RL) can be used to learn treatment policies and aid decision making in healthcare. However, given the need for generalization over complex state/action spaces, the incorporation of function approximators (e.g., deep neural networks) requires model selection to reduce overfitting and improve policy performance at deployment. Yet a standard validation pipeline for model selection requires running a learned policy in the actual environment, which is often infeasible in a healthcare setting. In this work, we investigate a model selection pipeline for offline RL that relies on off-policy evaluation (OPE) as a proxy for validation performance. We present an in-depth analysis of popular OPE methods, highlighting the additional hyperparameters and computational requirements (fitting/inference of auxiliary models) when used to rank a set of candidate policies. We compare the utility of different OPE methods as part of the model selection pipeline in the context of learning to treat patients with sepsis. Among all the OPE methods we considered, fitted Q evaluation (FQE) consistently leads to the best validation ranking, but at a high computational cost. To balance this trade-off between accuracy of ranking and computational efficiency, we propose a simple two-stage approach to accelerate model selection by avoiding potentially unnecessary computation. Our work serves as a practical guide for offline RL model selection and can help RL practitioners select policies using real-world datasets. To facilitate reproducibility and future extensions, the code accompanying this paper is available online.